[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:50 dist-test-slave-3t36 krb5kdc[2716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:50 dist-test-slave-3t36 krb5kdc[2716](info): set up 2 sockets
Sep 28 21:58:50 dist-test-slave-3t36 krb5kdc[2716](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 21:58:51.325392 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:58:51 dist-test-slave-3t36 krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:58:51.448305 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46377
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:46377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:51.531064 2729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:51.531309 2729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:51.531345 2729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:51.532480 2729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:51.532545 2729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:51.532569 2729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:51.532591 2729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:51.532618 2729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:51.534104 2729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:46377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:51.534389 2729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:51.534643 2729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:51.540242 2734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:51.540251 2737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:51.540287 2729 server_base.cc:1034] running on GCE node
W20260928 21:58:51.540251 2735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:51.540809 2729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:51.541133 2729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:51.542410 2729 hybrid_clock.cc:648] HybridClock initialized: now 1790632731542387 us; error 40 us; skew 500 ppm
Sep 28 21:58:51 dist-test-slave-3t36 krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:51.545030 2729 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:58:51.545446 2729 webserver.cc:460] Webserver started at http://127.2.165.126:39301/ using document root <none> and password file <none>
I20260928 21:58:51.545666 2729 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:51.545708 2729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:51.545796 2729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:51.546866 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "7c79d8d0e76a4430905a52e3dfe01d82"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "af12c1c0d7d04be1e7b1aec014383edb"
server_key_iv: "a224b2228ae83451eb43d26f7a86703a"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.547183 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "7c79d8d0e76a4430905a52e3dfe01d82"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "af12c1c0d7d04be1e7b1aec014383edb"
server_key_iv: "a224b2228ae83451eb43d26f7a86703a"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.548554 2729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:51.549316 2744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:51.549466 2729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:51.549546 2729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "7c79d8d0e76a4430905a52e3dfe01d82"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "af12c1c0d7d04be1e7b1aec014383edb"
server_key_iv: "a224b2228ae83451eb43d26f7a86703a"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.549613 2729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:51.559082 2729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:51.561365 2729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:51.561537 2729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:51.565404 2729 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:46377
I20260928 21:58:51.565487 2796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:46377 every 8 connection(s)
I20260928 21:58:51.565830 2729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:51.566763 2797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:51.569192 2797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82: Bootstrap starting.
I20260928 21:58:51.569789 2797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:51.570129 2797 log.cc:824] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:51.570928 2797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82: No bootstrap required, opened a new log
I20260928 21:58:51.572592 2797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46377 } }
I20260928 21:58:51.572816 2797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:51.572870 2797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c79d8d0e76a4430905a52e3dfe01d82, State: Initialized, Role: FOLLOWER
I20260928 21:58:51.573050 2797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46377 } }
I20260928 21:58:51.573149 2797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:51.573186 2797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:51.573237 2797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:51.573354 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 2729
I20260928 21:58:51.573457 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:58:51.573972 2797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46377 } }
I20260928 21:58:51.574086 2797 leader_election.cc:302] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c79d8d0e76a4430905a52e3dfe01d82; no voters:
I20260928 21:58:51.574237 2709 external_mini_cluster.cc:1428] Setting key 8538ebeafdfa61cbcd9b84ea3e1214f1
I20260928 21:58:51.574259 2797 leader_election.cc:288] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:51.574312 2800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:51.574462 2800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [term 1 LEADER]: Becoming Leader. State: Replica: 7c79d8d0e76a4430905a52e3dfe01d82, State: Running, Role: LEADER
I20260928 21:58:51.574529 2797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:51.574560 2800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46377 } }
I20260928 21:58:51.574860 2800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c79d8d0e76a4430905a52e3dfe01d82. Latest consensus state: current_term: 1 leader_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46377 } } }
I20260928 21:58:51.574932 2800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:51.575515 2805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:51.575769 2802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c79d8d0e76a4430905a52e3dfe01d82" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46377 } } }
I20260928 21:58:51.575866 2802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:51.576048 2805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:51.579130 2805 catalog_manager.cc:1347] Generated new cluster ID: 686d57582025486d9297097e20944541
I20260928 21:58:51.579195 2805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:51.587466 2805 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:51.588199 2805 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 21:58:51 dist-test-slave-3t36 krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:51.593734 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:51.575147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.2.165.126:46377)
0928 21:58:51.576856 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:51.576864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:51.576890 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:58:51.578729 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:51.578735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:51.578778 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:51.578924 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:51.581043 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.582575 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.584232 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.584959 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.585077 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:51.589713 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:51.589718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:51.589719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:51.589732 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:51.591702 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:51.592228 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:51.592231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:51.592232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:51.592336 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:51.592617 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:51.592618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:51.592618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:51.592895 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:51.593036 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:51.593344 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 21:58:51.593494 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":74,"threads_started":1}
I20260928 21:58:51.600049 2805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c79d8d0e76a4430905a52e3dfe01d82: Generated new TSK 0
I20260928 21:58:51.600279 2805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:58:51.632716 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46377
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:51.720647 2823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:51.720889 2823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:51.720916 2823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:51.722154 2823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:51.722203 2823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:51.722231 2823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:58:51.723740 2823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:51.723991 2823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:51.724241 2823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:58:51.729486 2823 server_base.cc:1034] running on GCE node
W20260928 21:58:51.729524 2829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:51.729458 2828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:51.729454 2831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:51.729914 2823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:51.730166 2823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:51.731325 2823 hybrid_clock.cc:648] HybridClock initialized: now 1790632731731304 us; error 33 us; skew 500 ppm
Sep 28 21:58:51 dist-test-slave-3t36 krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:51.733938 2823 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:58:51.734331 2823 webserver.cc:460] Webserver started at http://127.2.165.65:42657/ using document root <none> and password file <none>
I20260928 21:58:51.734513 2823 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:51.734553 2823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:51.734642 2823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:51.735698 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "9157a72e6b2442bea176f6c863da58c9"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "615c7682831e4b91046eb0932ad49931"
server_key_iv: "4bbc8c48401f825bbd4505fe61beb183"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.736084 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "9157a72e6b2442bea176f6c863da58c9"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "615c7682831e4b91046eb0932ad49931"
server_key_iv: "4bbc8c48401f825bbd4505fe61beb183"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.737565 2823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:51.738408 2838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:51.738577 2823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:58:51.738662 2823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "9157a72e6b2442bea176f6c863da58c9"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "615c7682831e4b91046eb0932ad49931"
server_key_iv: "4bbc8c48401f825bbd4505fe61beb183"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.738718 2823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:51.753479 2823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:51.755748 2823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:51.755869 2823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:51.756074 2823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:51.756330 2823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:51.756359 2823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:51.756393 2823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:51.756412 2823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:51.762426 2823 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:36629
I20260928 21:58:51.762566 2951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:36629 every 8 connection(s)
I20260928 21:58:51.762823 2823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:58:51.768270 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 2823
I20260928 21:58:51.768349 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:51.768515 2709 external_mini_cluster.cc:1428] Setting key 4b765ca8a93461bb2e449ab900feb31b
Sep 28 21:58:51 dist-test-slave-3t36 krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:51.773398 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:51.763635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:59255 (local address 127.2.165.126:46377)
0928 21:58:51.763693 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:51.763695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:51.763878 (+ 183us) server_negotiation.cc:407] Connection header received
0928 21:58:51.764490 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:51.764491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:51.764505 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:51.764583 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:51.765633 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.766141 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.767182 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.767406 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.767499 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:51.770202 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:51.770204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:51.770204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:51.770212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:51.771959 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:51.772378 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:51.772379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:51.772379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:51.772442 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:51.772717 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:51.772719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:51.772719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:51.772849 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:51.772926 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:51.773210 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 21:58:51.773308 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:58:51.773592 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:51.763594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46377 (local address 127.2.165.65:59255)
0928 21:58:51.763737 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:51.763786 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:51.764327 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:51.764665 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:51.764693 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:51.765304 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:51.765512 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:51.765517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.766232 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.766233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:51.767081 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:51.767083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.767508 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.767509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:51.767645 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:51.768214 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:51.768224 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:51.770098 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:51.772053 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:51.772060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:51.772063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:51.772293 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:51.772519 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:51.772520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:51.772520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:51.772629 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:51.772913 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:51.772920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:51.773072 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 21:58:51.773175 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 21:58:51.773340 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260928 21:58:51.774144 2952 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46377
I20260928 21:58:51.774278 2952 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:51.774461 2952 heartbeater.cc:511] Master 127.2.165.126:46377 requested a full tablet report, sending...
I20260928 21:58:51.775012 2761 ts_manager.cc:194] Registered new tserver with Master: 9157a72e6b2442bea176f6c863da58c9 (127.2.165.65:36629)
I20260928 21:58:51.775765 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:59255
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:58:51.796873 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46377
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:51.880350 2957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:51.880565 2957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:51.880589 2957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:51.881757 2957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:51.881801 2957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:51.881829 2957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:58:51.883410 2957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:51.883653 2957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:51.883850 2957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:51.889132 2962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:51.889111 2965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:51.889111 2963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:51.889144 2957 server_base.cc:1034] running on GCE node
I20260928 21:58:51.889537 2957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:51.889734 2957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:51.890923 2957 hybrid_clock.cc:648] HybridClock initialized: now 1790632731890910 us; error 36 us; skew 500 ppm
Sep 28 21:58:51 dist-test-slave-3t36 krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:51.893594 2957 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:58:51.894045 2957 webserver.cc:460] Webserver started at http://127.2.165.66:41827/ using document root <none> and password file <none>
I20260928 21:58:51.894284 2957 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:51.894351 2957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:51.894500 2957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:51.895570 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "af37ac9ad1574cd383669f164e0e5817"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "83d1e3ed3fb5c7a5451ab8b626082407"
server_key_iv: "bc60e1df93636f38029b672b19a651cf"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.895931 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "af37ac9ad1574cd383669f164e0e5817"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "83d1e3ed3fb5c7a5451ab8b626082407"
server_key_iv: "bc60e1df93636f38029b672b19a651cf"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.897927 2957 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 21:58:51.898838 2972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:51.899021 2957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:51.899127 2957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "af37ac9ad1574cd383669f164e0e5817"
format_stamp: "Formatted at 2026-09-28 21:58:51 on dist-test-slave-3t36"
server_key: "83d1e3ed3fb5c7a5451ab8b626082407"
server_key_iv: "bc60e1df93636f38029b672b19a651cf"
server_key_version: "encryptionkey@0"
I20260928 21:58:51.899204 2957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:51.917670 2957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:51.919915 2957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:51.920096 2957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:51.920346 2957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:51.920749 2957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:51.920799 2957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:51.920840 2957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:51.920888 2957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:51.927719 2957 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:44787
I20260928 21:58:51.927762 3085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:44787 every 8 connection(s)
I20260928 21:58:51.928100 2957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:51.931977 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 2957
I20260928 21:58:51.932062 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:51.932219 2709 external_mini_cluster.cc:1428] Setting key a9fbc9c7159fed8f6f30929c0c220e2d
Sep 28 21:58:51 dist-test-slave-3t36 krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:51.937994 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:51.928873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:44575 (local address 127.2.165.126:46377)
0928 21:58:51.928945 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:51.928947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:51.929140 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:58:51.929619 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:51.929620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:51.929630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:51.929682 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:51.930616 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.931108 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.932208 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.932515 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.932583 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:51.935044 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:51.935046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:51.935046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:51.935054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:51.936591 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:51.937019 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:51.937020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:51.937021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:51.937084 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:51.937333 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:51.937334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:51.937334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:51.937456 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:51.937532 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:51.937810 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 21:58:51.937899 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:58:51.938212 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:51.928874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46377 (local address 127.2.165.66:44575)
0928 21:58:51.929022 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:51.929059 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:51.929538 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:51.929738 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:51.929742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:51.930317 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:51.930531 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:51.930535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.931203 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.931204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:51.932116 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:51.932119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:51.932596 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:51.932597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:51.932773 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:51.933278 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:51.933289 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:51.934950 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:51.936701 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:51.936707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:51.936710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:51.936937 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:51.937144 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:51.937145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:51.937146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:51.937257 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:51.937522 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:51.937529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:51.937680 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 21:58:51.937789 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:58:51.937961 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260928 21:58:51.938706 3086 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46377
I20260928 21:58:51.938818 3086 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:51.939010 3086 heartbeater.cc:511] Master 127.2.165.126:46377 requested a full tablet report, sending...
I20260928 21:58:51.939414 2761 ts_manager.cc:194] Registered new tserver with Master: af37ac9ad1574cd383669f164e0e5817 (127.2.165.66:44787)
I20260928 21:58:51.939831 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:44575
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:58:51.959995 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46377
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:52.042070 3091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:52.042308 3091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:52.042346 3091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:52.043515 3091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:52.043581 3091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:52.043628 3091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:58:52.045244 3091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:52.045521 3091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:52.045753 3091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:58:52.050977 3091 server_base.cc:1034] running on GCE node
W20260928 21:58:52.050943 3099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:52.050935 3096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:52.050949 3097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:52.051414 3091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:52.051635 3091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:52.052753 3091 hybrid_clock.cc:648] HybridClock initialized: now 1790632732052732 us; error 37 us; skew 500 ppm
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:52.055552 3091 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:58:52.055995 3091 webserver.cc:460] Webserver started at http://127.2.165.67:40093/ using document root <none> and password file <none>
I20260928 21:58:52.056236 3091 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:52.056304 3091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:52.056455 3091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:52.057598 3091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "7972c8ad24bf40058003a3e263cfa0ab"
format_stamp: "Formatted at 2026-09-28 21:58:52 on dist-test-slave-3t36"
server_key: "4502d238616286335e119f8b5c75246e"
server_key_iv: "54dca479fbfad9a221af70baff5af4b5"
server_key_version: "encryptionkey@0"
I20260928 21:58:52.057991 3091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "7972c8ad24bf40058003a3e263cfa0ab"
format_stamp: "Formatted at 2026-09-28 21:58:52 on dist-test-slave-3t36"
server_key: "4502d238616286335e119f8b5c75246e"
server_key_iv: "54dca479fbfad9a221af70baff5af4b5"
server_key_version: "encryptionkey@0"
I20260928 21:58:52.059299 3091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:58:52.060071 3106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:52.060257 3091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:52.060355 3091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "7972c8ad24bf40058003a3e263cfa0ab"
format_stamp: "Formatted at 2026-09-28 21:58:52 on dist-test-slave-3t36"
server_key: "4502d238616286335e119f8b5c75246e"
server_key_iv: "54dca479fbfad9a221af70baff5af4b5"
server_key_version: "encryptionkey@0"
I20260928 21:58:52.060468 3091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:52.090186 3091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:52.092613 3091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:52.092825 3091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:52.093079 3091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:52.093412 3091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:52.093463 3091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:52.093509 3091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:52.093535 3091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:52.100157 3091 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:37117
I20260928 21:58:52.100206 3219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:37117 every 8 connection(s)
I20260928 21:58:52.100541 3091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:52.105136 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 3091
I20260928 21:58:52.105228 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:52.105399 2709 external_mini_cluster.cc:1428] Setting key 6f28f8124b48ac19743bb5a1765f0e44
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:52.110395 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:52.101313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:34201 (local address 127.2.165.126:46377)
0928 21:58:52.101366 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:52.101368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:52.101605 (+ 237us) server_negotiation.cc:407] Connection header received
0928 21:58:52.102078 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:52.102079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:52.102089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:52.102143 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:52.103143 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.103746 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.104757 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.104953 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.105016 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:52.107508 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:52.107510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:52.107510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:52.107518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:52.109016 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:52.109423 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:52.109424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:52.109425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:52.109485 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:52.109746 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:52.109747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:52.109747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:52.109865 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:52.109938 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:52.110214 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 21:58:52.110303 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:58:52.110590 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:52.101317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46377 (local address 127.2.165.67:34201)
0928 21:58:52.101481 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:52.101519 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:52.101992 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:52.102199 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:52.102202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:52.102847 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:52.103053 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:52.103058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.103822 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.103823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:52.104631 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:52.104633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.105036 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.105037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:52.105186 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:52.105756 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:52.105770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:52.107398 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:52.109109 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:52.109116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:52.109120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:52.109348 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:52.109563 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:52.109564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:52.109564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:52.109678 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:52.109927 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:52.109934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:52.110084 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 21:58:52.110192 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:58:52.110356 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 21:58:52.111016 3220 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46377
I20260928 21:58:52.111132 3220 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:52.111313 3220 heartbeater.cc:511] Master 127.2.165.126:46377 requested a full tablet report, sending...
I20260928 21:58:52.111763 2761 ts_manager.cc:194] Registered new tserver with Master: 7972c8ad24bf40058003a3e263cfa0ab (127.2.165.67:37117)
I20260928 21:58:52.112178 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:34201
I20260928 21:58:52.117842 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632732, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632732, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:58:52.141960 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:52.133856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.2.165.65:36629)
0928 21:58:52.134038 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:52.134043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:52.134054 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:58:52.134081 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:52.134083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:52.134096 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:52.134187 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:52.134800 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.135575 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.136381 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.136955 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.137017 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:52.139379 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:52.139387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:52.139387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:52.139395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:52.140915 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:52.141274 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:52.141277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:52.141277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:52.141333 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:52.141511 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:52.141512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:52.141512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:52.141656 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:52.141783 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:52.141818 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:58:52.141881 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
W20260928 21:58:52.142413 2866 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33744
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:58:52.167426 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:52.159763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.2.165.65:36629)
0928 21:58:52.159868 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:52.159871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:52.159999 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:58:52.160082 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:52.160083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:52.160099 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:52.160169 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:52.161136 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.161726 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.162489 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:52.162698 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:52.162790 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:52.165098 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:52.165100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:52.165100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:52.165109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:52.166331 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:52.166675 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:52.166676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:52.166676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:52.166731 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:52.166917 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:52.166918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:52.166918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:52.167025 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:52.167125 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:52.167230 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:58:52.167316 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:58:52.168000 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 2823
I20260928 21:58:52.173498 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 2957
I20260928 21:58:52.179523 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 3091
I20260928 21:58:52.184212 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 2729
2026-09-28T21:58:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1931 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[3234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[3234](info): set up 2 sockets
Sep 28 21:58:52 dist-test-slave-3t36 krb5kdc[3234](info): commencing operation
krb5kdc: starting...
W20260928 21:58:53.220363 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:58:53.348500 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:34369
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:34369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:53.429471 3247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:53.429737 3247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:53.429774 3247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:53.430928 3247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:53.430995 3247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:53.431018 3247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:53.431041 3247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:53.431066 3247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:53.432538 3247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:34369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:34369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:53.432839 3247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:53.433071 3247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:53.438112 3253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:53.438092 3252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:53.438093 3255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:53.438143 3247 server_base.cc:1034] running on GCE node
I20260928 21:58:53.438602 3247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:53.438927 3247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:53.440083 3247 hybrid_clock.cc:648] HybridClock initialized: now 1790632733440069 us; error 33 us; skew 500 ppm
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:53.442922 3247 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:58:53.443430 3247 webserver.cc:460] Webserver started at http://127.2.165.126:37327/ using document root <none> and password file <none>
I20260928 21:58:53.443701 3247 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:53.443770 3247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:53.443912 3247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:53.445096 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "4ca849986b8641cebfd0a0a2e0edbab0"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "b820f7e0c12b092acab41356a27da13a"
server_key_iv: "99144416dc18aecfb1c4020962e4a532"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.445456 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "4ca849986b8641cebfd0a0a2e0edbab0"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "b820f7e0c12b092acab41356a27da13a"
server_key_iv: "99144416dc18aecfb1c4020962e4a532"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.446827 3247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:53.447657 3262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.447885 3247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:53.448006 3247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "4ca849986b8641cebfd0a0a2e0edbab0"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "b820f7e0c12b092acab41356a27da13a"
server_key_iv: "99144416dc18aecfb1c4020962e4a532"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.448094 3247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:53.454756 3247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:53.457023 3247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:53.457178 3247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:53.461212 3247 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:34369
I20260928 21:58:53.461243 3314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:34369 every 8 connection(s)
I20260928 21:58:53.461598 3247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:53.462713 3315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:53.463277 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 3247
I20260928 21:58:53.463366 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:58:53.463536 2709 external_mini_cluster.cc:1428] Setting key 920addcaeb012300e09e397c88578b10
I20260928 21:58:53.465950 3315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0: Bootstrap starting.
I20260928 21:58:53.466562 3315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:53.466847 3315 log.cc:824] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:53.467518 3315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0: No bootstrap required, opened a new log
I20260928 21:58:53.469687 3315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34369 } }
I20260928 21:58:53.469853 3315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:53.469899 3315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca849986b8641cebfd0a0a2e0edbab0, State: Initialized, Role: FOLLOWER
I20260928 21:58:53.470048 3315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34369 } }
I20260928 21:58:53.470145 3315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:53.470180 3315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:53.470222 3315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:53.470947 3315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34369 } }
I20260928 21:58:53.471040 3315 leader_election.cc:302] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ca849986b8641cebfd0a0a2e0edbab0; no voters:
I20260928 21:58:53.471230 3315 leader_election.cc:288] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:53.471271 3320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:53.471426 3320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ca849986b8641cebfd0a0a2e0edbab0, State: Running, Role: LEADER
I20260928 21:58:53.471522 3315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:53.471619 3320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34369 } }
I20260928 21:58:53.471961 3320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ca849986b8641cebfd0a0a2e0edbab0. Latest consensus state: current_term: 1 leader_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34369 } } }
I20260928 21:58:53.471971 3321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca849986b8641cebfd0a0a2e0edbab0" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34369 } } }
I20260928 21:58:53.472052 3320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:53.472081 3321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:53.472813 3327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:53.473415 3327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:53.474957 3327 catalog_manager.cc:1347] Generated new cluster ID: b01dd0e83eaa4189ada9d680f2ce6e01
I20260928 21:58:53.475003 3327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:53.475260 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.464278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.2.165.126:34369)
0928 21:58:53.464702 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:53.464712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:53.464747 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:58:53.465272 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:53.465279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:53.465330 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:53.465482 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:53.466088 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.467092 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.467939 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.468379 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.468580 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.471198 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:53.471203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:53.471203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:53.471219 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:53.473585 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.474041 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.474044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.474045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.474144 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.474351 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.474353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.474353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.474677 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:53.474838 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:53.474961 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:58:53.475097 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":71,"threads_started":1}
I20260928 21:58:53.507465 3327 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:53.508315 3327 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:53.520509 3327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ca849986b8641cebfd0a0a2e0edbab0: Generated new TSK 0
I20260928 21:58:53.520774 3327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:58:53.552430 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34369
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:53.635900 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:53.636109 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:53.636134 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:53.637490 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:53.637535 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:53.637563 3341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:58:53.639087 3341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34369
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:53.639351 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:53.639613 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:53.645047 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:53.645095 3341 server_base.cc:1034] running on GCE node
W20260928 21:58:53.645031 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:53.645040 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:53.645457 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:53.645712 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:53.646885 3341 hybrid_clock.cc:648] HybridClock initialized: now 1790632733646872 us; error 27 us; skew 500 ppm
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:53.649643 3341 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:58:53.650058 3341 webserver.cc:460] Webserver started at http://127.2.165.65:33507/ using document root <none> and password file <none>
I20260928 21:58:53.650257 3341 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:53.650298 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:53.650395 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:53.651499 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "d33616c60dbd446fb2d6268946c4b7d8"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "5c4900b54537dffe826fab7f83a5b033"
server_key_iv: "082624bc61e6c76ed6d4e388fae07403"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.651810 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "d33616c60dbd446fb2d6268946c4b7d8"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "5c4900b54537dffe826fab7f83a5b033"
server_key_iv: "082624bc61e6c76ed6d4e388fae07403"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.653333 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:58:53.654098 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.654269 3341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:53.654357 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "d33616c60dbd446fb2d6268946c4b7d8"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "5c4900b54537dffe826fab7f83a5b033"
server_key_iv: "082624bc61e6c76ed6d4e388fae07403"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.654475 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:53.663733 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:53.666167 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:53.666344 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:53.666587 3341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:53.666901 3341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:53.666953 3341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.666994 3341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:53.667024 3341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.672978 3341 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:43629
I20260928 21:58:53.673067 3469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:43629 every 8 connection(s)
I20260928 21:58:53.673388 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:58:53.677624 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 3341
I20260928 21:58:53.677721 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:53.677875 2709 external_mini_cluster.cc:1428] Setting key 76632a9f6f1df5d4a8458155a98f9a19
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:53.684494 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.674303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:49809 (local address 127.2.165.126:34369)
0928 21:58:53.674370 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:53.674372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:53.674628 (+ 256us) server_negotiation.cc:407] Connection header received
0928 21:58:53.675560 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:53.675561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:53.675572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:53.675637 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:53.676262 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.676849 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.678272 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.678496 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.678590 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.681191 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:53.681193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:53.681193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:53.681201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:53.682986 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.683439 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.683441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.683441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.683512 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.683770 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.683771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.683772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.683940 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:53.684018 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:53.684270 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 21:58:53.684386 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:58:53.684581 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.674262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34369 (local address 127.2.165.65:49809)
0928 21:58:53.674491 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:53.674534 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:53.675462 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:53.675694 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:53.675697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:53.675960 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:53.676163 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:53.676167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.676958 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.676960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:53.678166 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:53.678169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.678592 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.678593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:53.678734 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.679307 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:53.679316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:53.681089 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:53.683092 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:53.683099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:53.683101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:53.683353 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:53.683587 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:53.683588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:53.683588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:53.683700 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:53.684012 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:53.684016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:53.684139 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 21:58:53.684226 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:58:53.684356 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
I20260928 21:58:53.685140 3470 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34369
I20260928 21:58:53.685256 3470 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:53.685441 3470 heartbeater.cc:511] Master 127.2.165.126:34369 requested a full tablet report, sending...
I20260928 21:58:53.686024 3279 ts_manager.cc:194] Registered new tserver with Master: d33616c60dbd446fb2d6268946c4b7d8 (127.2.165.65:43629)
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
I20260928 21:58:53.686770 3279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:49809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:58:53.704556 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34369
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:53.786676 3475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:53.786904 3475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:53.786931 3475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:53.788275 3475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:53.788322 3475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:53.788352 3475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:58:53.789862 3475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34369
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:53.790109 3475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:53.790333 3475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:53.795447 3481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:53.795424 3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:53.795414 3480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:53.795666 3475 server_base.cc:1034] running on GCE node
I20260928 21:58:53.795876 3475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:53.796100 3475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:53.797232 3475 hybrid_clock.cc:648] HybridClock initialized: now 1790632733797218 us; error 28 us; skew 500 ppm
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:53.799690 3475 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:58:53.800195 3475 webserver.cc:460] Webserver started at http://127.2.165.66:41277/ using document root <none> and password file <none>
I20260928 21:58:53.800420 3475 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:53.800485 3475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:53.800630 3475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:53.801687 3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "4abd672d14364579b1e7946a07e34a49"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "8f9dbac171e4a0ab9c797fb1e568fa25"
server_key_iv: "db1db497700239e883ceb1eec3abeb09"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.802050 3475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "4abd672d14364579b1e7946a07e34a49"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "8f9dbac171e4a0ab9c797fb1e568fa25"
server_key_iv: "db1db497700239e883ceb1eec3abeb09"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.803512 3475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:53.804281 3490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.804463 3475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:53.804557 3475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "4abd672d14364579b1e7946a07e34a49"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "8f9dbac171e4a0ab9c797fb1e568fa25"
server_key_iv: "db1db497700239e883ceb1eec3abeb09"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.804636 3475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:53.811929 3475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:53.813791 3475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:53.813942 3475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:53.814163 3475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:53.814461 3475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:53.814512 3475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.814546 3475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:53.814572 3475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.820446 3475 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:44469
I20260928 21:58:53.820504 3603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:44469 every 8 connection(s)
I20260928 21:58:53.820904 3475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:53.829483 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 3475
I20260928 21:58:53.829587 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:53.829747 2709 external_mini_cluster.cc:1428] Setting key a5b790eb5bce8a81b653559bcf42d00f
I20260928 21:58:53.830801 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.821773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:46395 (local address 127.2.165.126:34369)
0928 21:58:53.821855 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:53.821858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:53.821974 (+ 116us) server_negotiation.cc:407] Connection header received
0928 21:58:53.822902 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:53.822903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:53.822913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:53.822958 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:53.823609 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.824114 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.825290 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.825498 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.825570 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.827794 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:53.827796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:53.827796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:53.827803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:53.829260 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.829807 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.829809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.829810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.829871 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.830119 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.830120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.830120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.830239 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:53.830316 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:53.830616 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 21:58:53.830700 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:58:53.831000 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.821669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34369 (local address 127.2.165.66:46395)
0928 21:58:53.821864 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:53.821908 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:53.822814 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:53.823024 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:53.823028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:53.823284 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:53.823538 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:53.823543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.824200 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.824201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:53.825171 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:53.825174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.825609 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.825610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:53.825721 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.826290 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:53.826299 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:53.827688 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:53.829365 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:53.829371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:53.829373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:53.829709 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:53.829936 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:53.829937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:53.829937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:53.830050 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:53.830307 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:53.830313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:53.830462 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 21:58:53.830576 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:58:53.830750 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260928 21:58:53.831498 3604 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34369
I20260928 21:58:53.831591 3604 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:53.831808 3604 heartbeater.cc:511] Master 127.2.165.126:34369 requested a full tablet report, sending...
I20260928 21:58:53.832275 3279 ts_manager.cc:194] Registered new tserver with Master: 4abd672d14364579b1e7946a07e34a49 (127.2.165.66:44469)
I20260928 21:58:53.832754 3279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:46395
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:58:53.856652 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34369
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:53.940338 3609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:53.940585 3609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:53.940613 3609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:53.941931 3609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:53.941998 3609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:53.942029 3609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:58:53.943632 3609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34369
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:53.943917 3609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:53.944167 3609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:58:53.949745 3609 server_base.cc:1034] running on GCE node
W20260928 21:58:53.949777 3614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:53.949836 3615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:53.949841 3617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:53.950320 3609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:53.950614 3609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:53.951813 3609 hybrid_clock.cc:648] HybridClock initialized: now 1790632733951796 us; error 33 us; skew 500 ppm
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:53.954615 3609 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:58:53.955147 3609 webserver.cc:460] Webserver started at http://127.2.165.67:41457/ using document root <none> and password file <none>
I20260928 21:58:53.955384 3609 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:53.955461 3609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:53.955600 3609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:53.956775 3609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "df45a76a89cd4de09e7b48c9e1260473"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "cc6cd6d4e89bb71cda384eb2f768d9bf"
server_key_iv: "cf170b06a0d0e40252c5e66d4d3b02d3"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.957154 3609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "df45a76a89cd4de09e7b48c9e1260473"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "cc6cd6d4e89bb71cda384eb2f768d9bf"
server_key_iv: "cf170b06a0d0e40252c5e66d4d3b02d3"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.958702 3609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:53.959482 3624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.959700 3609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:53.959806 3609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "df45a76a89cd4de09e7b48c9e1260473"
format_stamp: "Formatted at 2026-09-28 21:58:53 on dist-test-slave-3t36"
server_key: "cc6cd6d4e89bb71cda384eb2f768d9bf"
server_key_iv: "cf170b06a0d0e40252c5e66d4d3b02d3"
server_key_version: "encryptionkey@0"
I20260928 21:58:53.959882 3609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:53.969466 3609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:53.971793 3609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:53.972084 3609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:53.972426 3609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:53.972906 3609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:53.972949 3609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.972972 3609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:53.972985 3609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:53.978881 3609 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:43467
I20260928 21:58:53.978971 3737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:43467 every 8 connection(s)
I20260928 21:58:53.979293 3609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:53.981711 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 3609
I20260928 21:58:53.981796 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:53.981966 2709 external_mini_cluster.cc:1428] Setting key e646fcfec2b19d36f0126498dd42f395
Sep 28 21:58:53 dist-test-slave-3t36 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:53.989351 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.980049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:44099 (local address 127.2.165.126:34369)
0928 21:58:53.980111 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:53.980114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:53.980354 (+ 240us) server_negotiation.cc:407] Connection header received
0928 21:58:53.981405 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:53.981406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:53.981419 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:53.981488 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:53.982267 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.983056 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.984051 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.984244 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.984303 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.986431 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:53.986432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:53.986433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:53.986439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:53.987910 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.988369 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.988370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.988371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.988439 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:53.988706 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:53.988707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:53.988707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:53.988831 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:53.988905 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:53.989174 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 21:58:53.989254 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:58:53.989553 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.980044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34369 (local address 127.2.165.67:44099)
0928 21:58:53.980230 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:53.980268 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:53.981289 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:53.981562 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:53.981566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:53.981890 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:53.982173 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:53.982180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.983147 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.983148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:53.983964 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:53.983967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.984316 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.984316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:53.984419 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.984967 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:53.984975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:53.986346 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:53.988032 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:53.988039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:53.988042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:53.988277 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:53.988504 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:53.988505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:53.988506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:53.988621 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:53.988897 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:53.988903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:53.989047 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 21:58:53.989153 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:58:53.989313 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260928 21:58:53.989986 3738 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34369
I20260928 21:58:53.990087 3738 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:53.990270 3738 heartbeater.cc:511] Master 127.2.165.126:34369 requested a full tablet report, sending...
I20260928 21:58:53.990655 3279 ts_manager.cc:194] Registered new tserver with Master: df45a76a89cd4de09e7b48c9e1260473 (127.2.165.67:43467)
I20260928 21:58:53.991042 3279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:44099
I20260928 21:58:53.993831 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:54.001590 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:53.995358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.2.165.126:34369)
0928 21:58:53.995440 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:53.995443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:53.995551 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:58:53.995645 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:53.995646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:53.995657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:53.995713 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:53.996243 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.996812 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.997534 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:53.997741 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:53.997813 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:53.999038 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:53.999039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:53.999040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:53.999047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:54.000305 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.000783 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.000784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.000785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.000850 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.001039 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.001040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.001040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.001195 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:54.001299 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.001402 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.001484 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:58:54.003597 3279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:54.005286 3279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:54.011855 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.007816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.2.165.66:44469)
0928 21:58:54.008163 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.008168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.008177 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:58:54.008216 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.008217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.008231 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.008440 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:54.008559 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.009673 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.010915 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.011422 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.011509 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.011571 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.011727 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.011769 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":68,"threads_started":1}
I20260928 21:58:54.011855 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.007495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:44469 (local address 127.0.0.1:34258)
0928 21:58:54.007666 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:54.007708 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:54.007789 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:54.008328 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:54.008329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:54.008331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:54.008368 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.008369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.009780 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.009781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.010842 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.010844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.011508 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.011509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.011605 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.011619 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:58:54.011712 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:58:54.011770 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":114,"threads_started":1}
I20260928 21:58:54.012172 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.007761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:43467 (local address 127.0.0.1:36172)
0928 21:58:54.007972 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:54.007983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:54.008034 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:54.008312 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:54.008313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:54.008316 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:54.008384 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.008385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.009512 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.009513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.010806 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.010808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.011789 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.011790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.011849 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.011853 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:54.011895 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 21:58:54.012100 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":42,"threads_started":1}
I20260928 21:58:54.012450 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.007971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.2.165.67:43467)
0928 21:58:54.008148 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.008152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.008162 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:54.008198 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.008200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.008218 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.008299 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:54.008610 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.009389 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.010910 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.011642 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.011709 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.011757 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.012332 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.012373 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":117,"threads_started":1}
I20260928 21:58:54.012652 3538 tablet_service.cc:1467] Processing CreateTablet for tablet ed5de6b7b44c4f89a61fb1226b84665e (DEFAULT_TABLE table=test-table [id=7fa4253e7bdb447ebe062c2a90b28027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:54.013084 3538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5de6b7b44c4f89a61fb1226b84665e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:54.013268 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.007606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.2.165.65:43629)
0928 21:58:54.007784 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.007790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.007955 (+ 165us) server_negotiation.cc:407] Connection header received
0928 21:58:54.008016 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.008019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.008035 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.008115 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:54.008378 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.009676 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.012100 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.012928 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.013009 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.013062 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.013153 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.013192 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260928 21:58:54.013185 3672 tablet_service.cc:1467] Processing CreateTablet for tablet ed5de6b7b44c4f89a61fb1226b84665e (DEFAULT_TABLE table=test-table [id=7fa4253e7bdb447ebe062c2a90b28027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:54.013460 3672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5de6b7b44c4f89a61fb1226b84665e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:54.013527 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.007500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:43629 (local address 127.0.0.1:51964)
0928 21:58:54.007893 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:54.007903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:54.007952 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:54.008218 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:54.008221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:54.008224 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:54.008300 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.008302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.010925 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.010926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.012005 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.012007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.013011 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.013012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.013074 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.013079 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:54.013214 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 21:58:54.013257 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260928 21:58:54.016688 3752 tablet_bootstrap.cc:492] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473: Bootstrap starting.
I20260928 21:58:54.017333 3404 tablet_service.cc:1467] Processing CreateTablet for tablet ed5de6b7b44c4f89a61fb1226b84665e (DEFAULT_TABLE table=test-table [id=7fa4253e7bdb447ebe062c2a90b28027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:54.017495 3752 tablet_bootstrap.cc:654] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:54.017719 3404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5de6b7b44c4f89a61fb1226b84665e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:54.018110 3752 log.cc:824] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:54.018504 3753 tablet_bootstrap.cc:492] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49: Bootstrap starting.
I20260928 21:58:54.019169 3753 tablet_bootstrap.cc:654] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:54.019770 3753 log.cc:824] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:54.020350 3752 tablet_bootstrap.cc:492] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473: No bootstrap required, opened a new log
I20260928 21:58:54.020440 3752 ts_tablet_manager.cc:1397] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:58:54.020570 3753 tablet_bootstrap.cc:492] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49: No bootstrap required, opened a new log
I20260928 21:58:54.020653 3753 ts_tablet_manager.cc:1397] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:54.021984 3756 tablet_bootstrap.cc:492] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8: Bootstrap starting.
I20260928 21:58:54.022451 3752 raft_consensus.cc:348] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.022616 3756 tablet_bootstrap.cc:654] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:54.022671 3752 raft_consensus.cc:374] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:54.022718 3752 raft_consensus.cc:729] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df45a76a89cd4de09e7b48c9e1260473, State: Initialized, Role: FOLLOWER
I20260928 21:58:54.022854 3752 consensus_queue.cc:260] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.023129 3752 ts_tablet_manager.cc:1428] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:54.023145 3756 log.cc:824] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:54.023134 3753 raft_consensus.cc:348] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.023273 3738 heartbeater.cc:503] Master 127.2.165.126:34369 was elected leader, sending a full tablet report...
I20260928 21:58:54.023399 3753 raft_consensus.cc:374] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:54.023461 3753 raft_consensus.cc:729] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4abd672d14364579b1e7946a07e34a49, State: Initialized, Role: FOLLOWER
I20260928 21:58:54.023586 3753 consensus_queue.cc:260] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.023910 3753 ts_tablet_manager.cc:1428] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:58:54.023934 3604 heartbeater.cc:503] Master 127.2.165.126:34369 was elected leader, sending a full tablet report...
I20260928 21:58:54.024609 3756 tablet_bootstrap.cc:492] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8: No bootstrap required, opened a new log
I20260928 21:58:54.024714 3756 ts_tablet_manager.cc:1397] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:58:54.026130 3756 raft_consensus.cc:348] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.026270 3756 raft_consensus.cc:374] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:54.026297 3756 raft_consensus.cc:729] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33616c60dbd446fb2d6268946c4b7d8, State: Initialized, Role: FOLLOWER
I20260928 21:58:54.026387 3756 consensus_queue.cc:260] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.026598 3756 ts_tablet_manager.cc:1428] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:54.026636 3470 heartbeater.cc:503] Master 127.2.165.126:34369 was elected leader, sending a full tablet report...
I20260928 21:58:54.068043 3757 raft_consensus.cc:482] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:54.068179 3757 raft_consensus.cc:504] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.068501 3757 leader_election.cc:288] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4abd672d14364579b1e7946a07e34a49 (127.2.165.66:44469), d33616c60dbd446fb2d6268946c4b7d8 (127.2.165.65:43629)
W20260928 21:58:54.071470 3605 tablet.cc:2369] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:54.071602 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.068752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:44469 (local address 127.2.165.67:38887)
0928 21:58:54.068803 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:54.068818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:54.068889 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:54.069090 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:54.069091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:54.069093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:54.069192 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.069194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.070006 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.070007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.070766 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.070768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.071330 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.071331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.071412 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.071415 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:54.071473 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:58:54.071518 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:58:54.071602 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.068889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:38887 (local address 127.2.165.66:44469)
0928 21:58:54.068940 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.068942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.068951 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:58:54.068978 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.068978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.068988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.069063 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:54.069265 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.069933 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.070856 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.071250 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.071334 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.071343 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.071492 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.071528 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:58:54.071947 3558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5de6b7b44c4f89a61fb1226b84665e" candidate_uuid: "df45a76a89cd4de09e7b48c9e1260473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4abd672d14364579b1e7946a07e34a49" is_pre_election: true
I20260928 21:58:54.072124 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.068809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:43629 (local address 127.2.165.67:60993)
0928 21:58:54.069008 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:54.069018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:54.069106 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:54.069284 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:54.069285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:54.069287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:54.069366 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.069367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.070183 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.070184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.071018 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:54.071020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.071826 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.071827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:54.071925 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.071928 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:54.072014 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:58:54.072053 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":130,"threads_started":1}
I20260928 21:58:54.072132 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.068819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:60993 (local address 127.2.165.65:43629)
0928 21:58:54.068880 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.068882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.069107 (+ 225us) server_negotiation.cc:407] Connection header received
0928 21:58:54.069172 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.069173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.069182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.069278 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:54.069446 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.070104 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.071101 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.071612 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.071704 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.071713 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.072012 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.072068 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:58:54.072124 3558 raft_consensus.cc:2454] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df45a76a89cd4de09e7b48c9e1260473 in term 0.
I20260928 21:58:54.072379 3628 leader_election.cc:302] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abd672d14364579b1e7946a07e34a49, df45a76a89cd4de09e7b48c9e1260473; no voters:
I20260928 21:58:54.072490 3424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5de6b7b44c4f89a61fb1226b84665e" candidate_uuid: "df45a76a89cd4de09e7b48c9e1260473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33616c60dbd446fb2d6268946c4b7d8" is_pre_election: true
I20260928 21:58:54.072556 3757 raft_consensus.cc:2781] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:54.072602 3757 raft_consensus.cc:482] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:54.072628 3757 raft_consensus.cc:3037] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:54.072634 3424 raft_consensus.cc:2454] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df45a76a89cd4de09e7b48c9e1260473 in term 0.
I20260928 21:58:54.073342 3757 raft_consensus.cc:504] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.073484 3757 leader_election.cc:288] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [CANDIDATE]: Term 1 election: Requested vote from peers 4abd672d14364579b1e7946a07e34a49 (127.2.165.66:44469), d33616c60dbd446fb2d6268946c4b7d8 (127.2.165.65:43629)
I20260928 21:58:54.073629 3424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5de6b7b44c4f89a61fb1226b84665e" candidate_uuid: "df45a76a89cd4de09e7b48c9e1260473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33616c60dbd446fb2d6268946c4b7d8"
I20260928 21:58:54.073657 3558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5de6b7b44c4f89a61fb1226b84665e" candidate_uuid: "df45a76a89cd4de09e7b48c9e1260473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4abd672d14364579b1e7946a07e34a49"
I20260928 21:58:54.073719 3558 raft_consensus.cc:3037] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:54.073709 3424 raft_consensus.cc:3037] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:54.074596 3424 raft_consensus.cc:2454] T ed5de6b7b44c4f89a61fb1226b84665e P d33616c60dbd446fb2d6268946c4b7d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df45a76a89cd4de09e7b48c9e1260473 in term 1.
I20260928 21:58:54.074597 3558 raft_consensus.cc:2454] T ed5de6b7b44c4f89a61fb1226b84665e P 4abd672d14364579b1e7946a07e34a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df45a76a89cd4de09e7b48c9e1260473 in term 1.
I20260928 21:58:54.074800 3627 leader_election.cc:302] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d33616c60dbd446fb2d6268946c4b7d8, df45a76a89cd4de09e7b48c9e1260473; no voters:
I20260928 21:58:54.074901 3757 raft_consensus.cc:2781] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:54.075088 3757 raft_consensus.cc:686] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [term 1 LEADER]: Becoming Leader. State: Replica: df45a76a89cd4de09e7b48c9e1260473, State: Running, Role: LEADER
I20260928 21:58:54.075191 3757 consensus_queue.cc:237] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } }
I20260928 21:58:54.076057 3279 catalog_manager.cc:5495] T ed5de6b7b44c4f89a61fb1226b84665e P df45a76a89cd4de09e7b48c9e1260473 reported cstate change: term changed from 0 to 1, leader changed from <none> to df45a76a89cd4de09e7b48c9e1260473 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "df45a76a89cd4de09e7b48c9e1260473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abd672d14364579b1e7946a07e34a49" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33616c60dbd446fb2d6268946c4b7d8" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df45a76a89cd4de09e7b48c9e1260473" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43467 } health_report { overall_health: HEALTHY } } }
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:58:54.094751 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.086140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.2.165.65:43629)
0928 21:58:54.086243 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.086245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.086358 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:58:54.086449 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.086450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.086461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.086514 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:54.087174 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.087676 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.089011 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.089216 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.089303 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.091734 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:54.091748 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:54.091748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:54.091756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:54.093528 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.093880 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.093885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.093885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.094075 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.094256 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.094256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.094257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.094411 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:54.094516 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.094600 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.094681 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:58:54.118710 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.111194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.2.165.65:43629)
0928 21:58:54.111278 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.111280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.111363 (+ 83us) server_negotiation.cc:407] Connection header received
0928 21:58:54.111457 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.111458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.111530 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.111593 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:54.112234 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.112801 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.113565 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.113772 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.113855 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.116116 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:54.116118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:54.116118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:54.116125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:54.117702 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.118019 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.118020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.118020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.118071 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.118226 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.118226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.118227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.118317 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:54.118384 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.118537 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.118637 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:58:54.142356 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:54.134654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.2.165.65:43629)
0928 21:58:54.134754 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:54.134756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:54.134770 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:58:54.134841 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:54.134842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:54.134852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:54.134908 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:54.135468 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.135971 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.136763 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:54.136960 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:54.137024 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:54.139602 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:54.139605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:54.139605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:54.139615 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:54.141179 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.141570 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.141572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.141572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.141632 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:54.141831 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:54.141832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:54.141833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:54.141956 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:54.142063 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:54.142162 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 21:58:54.142248 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 21:58:54.142941 3384 tablet_service.cc:2908] Rejecting scan request for tablet ed5de6b7b44c4f89a61fb1226b84665e: Uninitialized: clean time has not yet been initialized
I20260928 21:58:54.143880 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 3341
I20260928 21:58:54.149181 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 3475
I20260928 21:58:54.153743 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 3609
I20260928 21:58:54.157846 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 3247
2026-09-28T21:58:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1974 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3776](info): set up 2 sockets
Sep 28 21:58:54 dist-test-slave-3t36 krb5kdc[3776](info): commencing operation
krb5kdc: starting...
W20260928 21:58:55.228296 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:58:55.352454 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45689
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:55.437544 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:55.437820 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:55.437858 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:55.439128 3789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:55.439201 3789 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:58:55.439224 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:55.439248 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:55.439286 3789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:55.439312 3789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:55.441008 3789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45689
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:55.441335 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:55.441597 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:55.446877 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:55.446877 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:55.446930 3789 server_base.cc:1034] running on GCE node
W20260928 21:58:55.446861 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:55.447376 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:55.447731 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:55.448942 3789 hybrid_clock.cc:648] HybridClock initialized: now 1790632735448940 us; error 48 us; skew 500 ppm
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:55.468786 3789 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:58:55.469491 3789 webserver.cc:460] Webserver started at http://127.2.165.126:33541/ using document root <none> and password file <none>
I20260928 21:58:55.469821 3789 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:55.469890 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:55.470105 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:55.471804 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "c13dd4fc0629460da948e1a29342b395"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "af33eb9db3060b04682a5380f7c74470"
server_key_iv: "00f4a3602f56c9743445a6da71c615aa"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.472271 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "c13dd4fc0629460da948e1a29342b395"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "af33eb9db3060b04682a5380f7c74470"
server_key_iv: "00f4a3602f56c9743445a6da71c615aa"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.473937 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 21:58:55.474877 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:55.475157 3789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:55.475277 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "c13dd4fc0629460da948e1a29342b395"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "af33eb9db3060b04682a5380f7c74470"
server_key_iv: "00f4a3602f56c9743445a6da71c615aa"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.475380 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:55.491634 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:55.494318 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:55.494498 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:55.499626 3789 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45689
I20260928 21:58:55.499657 3866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45689 every 8 connection(s)
I20260928 21:58:55.500087 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:55.500881 3867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:55.503259 3867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395: Bootstrap starting.
I20260928 21:58:55.503857 3867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:55.504113 3867 log.cc:824] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:55.505004 3867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395: No bootstrap required, opened a new log
I20260928 21:58:55.507534 3867 raft_consensus.cc:348] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13dd4fc0629460da948e1a29342b395" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45689 } }
I20260928 21:58:55.507723 3867 raft_consensus.cc:374] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:55.507751 3867 raft_consensus.cc:729] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13dd4fc0629460da948e1a29342b395, State: Initialized, Role: FOLLOWER
I20260928 21:58:55.507928 3867 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13dd4fc0629460da948e1a29342b395" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45689 } }
I20260928 21:58:55.508024 3867 raft_consensus.cc:388] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:55.508052 3867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:55.508044 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 3789
I20260928 21:58:55.508126 3867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:55.508149 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:58:55.508308 2709 external_mini_cluster.cc:1428] Setting key 8519c1b7992c212e420079aadded6e5a
I20260928 21:58:55.508924 3867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13dd4fc0629460da948e1a29342b395" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45689 } }
I20260928 21:58:55.509068 3867 leader_election.cc:302] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c13dd4fc0629460da948e1a29342b395; no voters:
I20260928 21:58:55.509241 3867 leader_election.cc:288] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:55.509282 3872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:55.509428 3872 raft_consensus.cc:686] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [term 1 LEADER]: Becoming Leader. State: Replica: c13dd4fc0629460da948e1a29342b395, State: Running, Role: LEADER
I20260928 21:58:55.509546 3867 sys_catalog.cc:564] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:55.509528 3872 consensus_queue.cc:237] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13dd4fc0629460da948e1a29342b395" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45689 } }
I20260928 21:58:55.509948 3872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c13dd4fc0629460da948e1a29342b395. Latest consensus state: current_term: 1 leader_uuid: "c13dd4fc0629460da948e1a29342b395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13dd4fc0629460da948e1a29342b395" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45689 } } }
I20260928 21:58:55.510059 3872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:55.509943 3873 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c13dd4fc0629460da948e1a29342b395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13dd4fc0629460da948e1a29342b395" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45689 } } }
I20260928 21:58:55.510479 3873 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:55.510658 3878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:55.511264 3878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:55.513015 3878 catalog_manager.cc:1347] Generated new cluster ID: a8aa1ccf12f24aab8c51b17408c1b670
I20260928 21:58:55.513078 3878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:55.519380 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.508829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.2.165.126:45689)
0928 21:58:55.509012 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:55.509023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:55.509063 (+ 40us) server_negotiation.cc:407] Connection header received
0928 21:58:55.509220 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:55.509228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:55.509296 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:55.510361 (+ 1065us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:55.510503 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.511942 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.513024 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.513513 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.513632 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.515975 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:55.515980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:55.515980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:55.515993 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:55.517783 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.518160 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.518163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.518164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.518275 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.518500 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.518501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.518502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.518763 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:55.518946 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:55.519016 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:58:55.519162 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260928 21:58:55.521875 3878 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:55.522473 3878 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:55.526852 3878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395: Generated new TSK 0
I20260928 21:58:55.527120 3878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:58:55.561348 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45689
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:55.645473 3893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:55.645726 3893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:55.645776 3893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:55.647020 3893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:55.647094 3893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:55.647128 3893 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:58:55.647176 3893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:58:55.648710 3893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:45689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:55.648990 3893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:55.649235 3893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:55.654421 3899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:55.654418 3898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:55.654417 3901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:55.654448 3893 server_base.cc:1034] running on GCE node
I20260928 21:58:55.654843 3893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:55.655067 3893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:55.656237 3893 hybrid_clock.cc:648] HybridClock initialized: now 1790632735656225 us; error 38 us; skew 500 ppm
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:55.658969 3893 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:58:55.659368 3893 webserver.cc:460] Webserver started at http://127.2.165.65:35319/ using document root <none> and password file <none>
I20260928 21:58:55.659621 3893 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:55.659667 3893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:55.659799 3893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:55.660938 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "eabcce7f6b8c4cada79ae523c95840ef"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "7cc723d217654a1f44c3fb6c5db2dd64"
server_key_iv: "2db3972fe3121b0ebef0b329a6e49998"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.661290 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "eabcce7f6b8c4cada79ae523c95840ef"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "7cc723d217654a1f44c3fb6c5db2dd64"
server_key_iv: "2db3972fe3121b0ebef0b329a6e49998"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.662669 3893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:55.663482 3908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:55.663656 3893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:55.663769 3893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "eabcce7f6b8c4cada79ae523c95840ef"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "7cc723d217654a1f44c3fb6c5db2dd64"
server_key_iv: "2db3972fe3121b0ebef0b329a6e49998"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.663849 3893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:55.680649 3893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:55.683180 3893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:55.683351 3893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:55.683965 3893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:55.684031 3893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:55.684077 3893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:55.684106 3893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:55.691743 3893 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:39563
I20260928 21:58:55.691850 4023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:39563 every 8 connection(s)
I20260928 21:58:55.692169 3893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:58:55.695235 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.684560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:57629 (local address 127.2.165.126:45689)
0928 21:58:55.684608 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:55.684611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:55.684891 (+ 280us) server_negotiation.cc:407] Connection header received
0928 21:58:55.685451 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:55.685453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:55.685470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:55.685530 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:55.686257 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.686902 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.688014 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.688233 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.688318 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.692067 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:55.692069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:55.692070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:55.692085 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:55.693783 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.694214 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.694215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.694215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.694284 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.694579 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.694580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.694581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.694755 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:55.694820 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:55.695086 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 21:58:55.695169 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:58:55.695433 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.684531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45689 (local address 127.2.165.65:57629)
0928 21:58:55.684743 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:55.684792 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:55.685334 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:55.685610 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:55.685615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:55.685895 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:55.686141 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.686146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.687001 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.687002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.687915 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.687917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.688331 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.688332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.688505 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.689175 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:55.689187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:55.691955 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:55.693873 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.693879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.693881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.694123 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.694369 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.694370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.694370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.694487 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.694824 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:55.694829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:55.694957 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 21:58:55.695058 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:58:55.695191 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260928 21:58:55.696080 4024 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45689
I20260928 21:58:55.696182 4024 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:55.696362 4024 heartbeater.cc:511] Master 127.2.165.126:45689 requested a full tablet report, sending...
I20260928 21:58:55.696575 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 3893
I20260928 21:58:55.696714 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:55.696914 2709 external_mini_cluster.cc:1428] Setting key 56ed09f83d4f60356ee9d1467798f74e
I20260928 21:58:55.697021 3820 ts_manager.cc:194] Registered new tserver with Master: eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563)
I20260928 21:58:55.697997 3820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:57629
I20260928 21:58:55.703604 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.697206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45689 (local address 127.2.165.65:43781)
0928 21:58:55.697333 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:55.697345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:55.697407 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:55.697537 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:55.697538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:55.697724 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:55.697799 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.697801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.698661 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.698662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.699207 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.699209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.699680 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.699682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.699766 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.700234 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:55.700240 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:55.700729 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:55.702633 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.702634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.702634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.702800 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.702992 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.702993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.702993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.703043 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.703338 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:55.703339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:55.703391 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 21:58:55.703447 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:58:55.703528 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":113,"threads_started":1}
I20260928 21:58:55.703711 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.697270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:43781 (local address 127.2.165.126:45689)
0928 21:58:55.697309 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:55.697310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:55.697393 (+ 83us) server_negotiation.cc:407] Connection header received
0928 21:58:55.697476 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:55.697477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:55.697485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:55.697525 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:55.697866 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.698567 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.699319 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.699548 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.699617 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.700805 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:55.700807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:55.700807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:55.700815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:55.702538 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.702867 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.702868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.702869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.702926 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.703104 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.703105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.703106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.703262 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:55.703339 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:55.703494 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 21:58:55.703578 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:58:55.726059 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45689
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:55.810583 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:55.810840 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:55.810868 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:55.812073 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:55.812119 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:55.812141 4033 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:58:55.812160 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:58:55.813580 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:45689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:55.813779 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:55.813963 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:55.819082 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:55.819126 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:55.819135 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:55.819325 4033 server_base.cc:1034] running on GCE node
I20260928 21:58:55.819564 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:55.819834 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:55.820992 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790632735820989 us; error 44 us; skew 500 ppm
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:55.823752 4033 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:58:55.824259 4033 webserver.cc:460] Webserver started at http://127.2.165.66:39349/ using document root <none> and password file <none>
I20260928 21:58:55.824493 4033 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:55.824568 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:55.824745 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:55.825933 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "3d730df97f26428493b5ecb5db023f3d"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "3766d6a3d4953755a36100a609632438"
server_key_iv: "a758df13ff49291fcbc63fcca60324e6"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.826310 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "3d730df97f26428493b5ecb5db023f3d"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "3766d6a3d4953755a36100a609632438"
server_key_iv: "a758df13ff49291fcbc63fcca60324e6"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.827752 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:55.828603 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:55.828858 4033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:55.828948 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "3d730df97f26428493b5ecb5db023f3d"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "3766d6a3d4953755a36100a609632438"
server_key_iv: "a758df13ff49291fcbc63fcca60324e6"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.829020 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:55.839802 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:55.842085 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:55.842209 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:55.842692 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:55.842729 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:55.842753 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:55.842767 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:55.850111 4033 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:34809
I20260928 21:58:55.850194 4163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:34809 every 8 connection(s)
I20260928 21:58:55.850508 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:55.851589 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 4033
I20260928 21:58:55.851698 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:55.851873 2709 external_mini_cluster.cc:1428] Setting key 1d4cfc89febf1d7f894b2a8c23490e12
I20260928 21:58:55.853156 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.843337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:59775 (local address 127.2.165.126:45689)
0928 21:58:55.843405 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:55.843408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:55.843630 (+ 222us) server_negotiation.cc:407] Connection header received
0928 21:58:55.844262 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:55.844264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:55.844279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:55.844363 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:55.845111 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.845630 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.846683 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.846983 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.847042 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.849862 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:55.849864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:55.849865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:55.849874 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:55.851543 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.852006 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.852007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.852008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.852077 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.852396 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.852397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.852397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.852537 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:55.852601 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:55.852926 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 21:58:55.853016 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:58:55.853186 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.843327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45689 (local address 127.2.165.66:59775)
0928 21:58:55.843491 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:55.843534 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:55.844115 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:55.844445 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:55.844451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:55.844809 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:55.845002 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.845008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.845725 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.845726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.846567 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.846570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.847079 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.847080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.847243 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.847821 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:55.847829 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:55.849733 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:55.851648 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.851654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.851656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.851918 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.852172 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.852173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.852173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.852330 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.852602 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:55.852608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:55.852790 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 21:58:55.852879 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:58:55.853010 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260928 21:58:55.853652 4164 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45689
I20260928 21:58:55.853732 4164 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:55.853929 4164 heartbeater.cc:511] Master 127.2.165.126:45689 requested a full tablet report, sending...
I20260928 21:58:55.854732 3821 ts_manager.cc:194] Registered new tserver with Master: 3d730df97f26428493b5ecb5db023f3d (127.2.165.66:34809)
I20260928 21:58:55.855247 3821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:59775
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 21:58:55.862679 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.854719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:38787 (local address 127.2.165.126:45689)
0928 21:58:55.854758 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:55.854760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:55.854953 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:58:55.855026 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:55.855027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:55.855038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:55.855085 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:55.856792 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.857290 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.858536 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.858736 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.858798 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.859951 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:55.859953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:55.859953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:55.859960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:55.861478 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.861845 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.861846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.861847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.861929 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:55.862187 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:55.862187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:55.862188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:55.862335 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:55.862399 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:55.862509 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 21:58:55.862614 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:58:55.862686 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:55.854764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45689 (local address 127.2.165.66:38787)
0928 21:58:55.854879 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:55.854889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:55.854968 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:55.855087 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:55.855088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:55.855277 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:55.855326 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.855328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.857402 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.857404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.858435 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:55.858439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:55.858815 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:55.858816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:55.858902 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:55.859427 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:55.859432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:55.859849 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:55.861579 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.861580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.861581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.861773 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.862029 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:55.862030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:55.862030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:55.862090 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:55.862401 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:55.862402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:55.862442 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:58:55.862494 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:58:55.862580 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:58:55.880538 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45689
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:55.963816 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:55.964057 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:55.964083 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:55.965278 4173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:55.965319 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:55.965340 4173 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:58:55.965358 4173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:58:55.966843 4173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:45689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:55.967087 4173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:55.967301 4173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:55.972772 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:55.972781 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:55.972834 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:55.973035 4173 server_base.cc:1034] running on GCE node
I20260928 21:58:55.973232 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:55.973459 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:55.974622 4173 hybrid_clock.cc:648] HybridClock initialized: now 1790632735974605 us; error 29 us; skew 500 ppm
Sep 28 21:58:55 dist-test-slave-3t36 krb5kdc[3776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:55.977315 4173 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:58:55.977885 4173 webserver.cc:460] Webserver started at http://127.2.165.67:39329/ using document root <none> and password file <none>
I20260928 21:58:55.978135 4173 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:55.978188 4173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:55.978341 4173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:55.979503 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "1e4c36c95d2d442f995e8f5646fe6d61"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "7a584fb78765d57899e9839b869bd60c"
server_key_iv: "efb5f6c952410f5673b3d7b45e262219"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.979863 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4c36c95d2d442f995e8f5646fe6d61"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "7a584fb78765d57899e9839b869bd60c"
server_key_iv: "efb5f6c952410f5673b3d7b45e262219"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.981292 4173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:55.982090 4188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:55.982259 4173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:55.982345 4173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "1e4c36c95d2d442f995e8f5646fe6d61"
format_stamp: "Formatted at 2026-09-28 21:58:55 on dist-test-slave-3t36"
server_key: "7a584fb78765d57899e9839b869bd60c"
server_key_iv: "efb5f6c952410f5673b3d7b45e262219"
server_key_version: "encryptionkey@0"
I20260928 21:58:55.982402 4173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:55.999008 4173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:56.001430 4173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:56.001552 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:56.002028 4173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:56.002066 4173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:56.002092 4173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:56.002142 4173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[3776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:56.009410 4173 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:33225
I20260928 21:58:56.009485 4303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:33225 every 8 connection(s)
I20260928 21:58:56.009819 4173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:56.012979 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.002626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:51967 (local address 127.2.165.126:45689)
0928 21:58:56.002681 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.002683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.002906 (+ 223us) server_negotiation.cc:407] Connection header received
0928 21:58:56.003496 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.003498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.003514 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.003596 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:56.004544 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.005433 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.006598 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.006924 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.007028 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.010007 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:56.010008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:56.010008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:56.010034 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:56.011553 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.012071 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.012072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.012072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.012126 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.012350 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.012350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.012351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.012457 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:56.012507 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.012838 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.012914 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:58:56.013140 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.002609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45689 (local address 127.2.165.67:51967)
0928 21:58:56.002783 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.002820 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.003349 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.003689 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.003695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.004093 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:56.004425 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.004432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.005595 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.005596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.006490 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.006492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.007054 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.007055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.007265 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.007847 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:56.007858 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:56.009887 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:56.011689 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:56.011695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:56.011699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:56.011970 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:56.012186 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:56.012187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:56.012187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:56.012296 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:56.012507 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:56.012511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:56.012719 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.012820 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.012950 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260928 21:58:56.013628 4304 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45689
I20260928 21:58:56.013712 4304 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:56.013903 4304 heartbeater.cc:511] Master 127.2.165.126:45689 requested a full tablet report, sending...
I20260928 21:58:56.014307 3820 ts_manager.cc:194] Registered new tserver with Master: 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67:33225)
I20260928 21:58:56.014839 3820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:51967
I20260928 21:58:56.015874 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 4173
I20260928 21:58:56.015969 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:56.016135 2709 external_mini_cluster.cc:1428] Setting key 5072659dad4fff52b3c3a9b1acb1fc26
I20260928 21:58:56.017130 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:56.021960 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.014668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45689 (local address 127.2.165.67:56191)
0928 21:58:56.014825 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.014835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.014932 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.015126 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.015126 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.015296 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:56.015347 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.015348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.016312 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.016313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.017129 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.017131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.017572 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.017573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.017637 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.018165 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:56.018169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:56.018470 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:56.020755 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:56.020757 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:56.020757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:56.020945 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:56.021177 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:56.021178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:56.021178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:56.021247 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:56.021570 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:56.021571 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:56.021621 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.021688 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.021854 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260928 21:58:56.021977 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.014731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:56191 (local address 127.2.165.126:45689)
0928 21:58:56.014767 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.014769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.014887 (+ 118us) server_negotiation.cc:407] Connection header received
0928 21:58:56.015008 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.015009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.015045 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.015077 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:56.015417 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.016150 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.017205 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.017481 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.017555 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.018562 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:56.018563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:56.018564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:56.018571 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:56.020637 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.021020 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.021022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.021022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.021086 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.021315 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.021316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.021317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.021484 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:56.021557 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.021734 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.021826 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[3776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632736, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[3776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632736, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:56.053030 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.043712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.2.165.126:45689)
0928 21:58:56.043791 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.043794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.043810 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:58:56.043849 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.043850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.043868 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.043963 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:56.044949 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.045869 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.047170 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.047376 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.047439 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.050275 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:56.050277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:56.050278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:56.050292 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:56.051804 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.052195 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.052197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.052197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.052257 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.052452 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.052454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.052454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.052579 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:56.052707 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.052831 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.052919 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:58:56.055022 3820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:56.056772 3820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:56.063859 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.059786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.2.165.65:39563)
0928 21:58:56.060164 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.060169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.060179 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:56.060218 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.060221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.060238 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.060317 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:56.060453 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.061624 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.062890 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.063385 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.063481 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.063543 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.063708 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.063751 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":70,"threads_started":1}
I20260928 21:58:56.063854 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.059680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39563 (local address 127.0.0.1:39068)
0928 21:58:56.059969 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.059985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.060027 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.060313 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.060314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.060316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:56.060367 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.060369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.061730 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.061731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.062768 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.062770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.063484 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.063485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.063570 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.063585 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.063688 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.063747 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":77,"threads_started":1}
I20260928 21:58:56.064061 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.059591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.2.165.67:33225)
0928 21:58:56.059810 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.059815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.059825 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:56.059866 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.059868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.059883 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.059958 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:56.060249 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.061248 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.062668 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.063412 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.063481 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.063539 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.063905 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.063953 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
I20260928 21:58:56.064239 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.059958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.2.165.66:34809)
0928 21:58:56.060216 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.060221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.060473 (+ 252us) server_negotiation.cc:407] Connection header received
0928 21:58:56.060548 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.060551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.060567 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.060657 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:56.060847 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.061853 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.063063 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.063793 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.063854 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.063904 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.064092 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.064134 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260928 21:58:56.064177 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.059850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:34809 (local address 127.0.0.1:34610)
0928 21:58:56.060410 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.060421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.060471 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.060728 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.060729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.060731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:56.060770 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.060772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.061938 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.061939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.062962 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.062965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.063937 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.063937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.063993 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.063997 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.064049 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.064097 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":69,"threads_started":1}
I20260928 21:58:56.063866 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.059453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:33225 (local address 127.0.0.1:45798)
0928 21:58:56.059606 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.059663 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.059737 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.060039 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.060043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.060046 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:56.060122 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.060157 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.061327 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.061328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.062569 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.062571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.063574 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.063575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.063651 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.063656 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.063708 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.063754 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260928 21:58:56.065275 3958 tablet_service.cc:1467] Processing CreateTablet for tablet b681911790a145a8b42123631b5d5a31 (DEFAULT_TABLE table=test-table [id=4bf59173ae114d31aee6a52519cc2b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:56.065284 4098 tablet_service.cc:1467] Processing CreateTablet for tablet b681911790a145a8b42123631b5d5a31 (DEFAULT_TABLE table=test-table [id=4bf59173ae114d31aee6a52519cc2b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:56.065321 4238 tablet_service.cc:1467] Processing CreateTablet for tablet b681911790a145a8b42123631b5d5a31 (DEFAULT_TABLE table=test-table [id=4bf59173ae114d31aee6a52519cc2b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:56.065644 3958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b681911790a145a8b42123631b5d5a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.065644 4238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b681911790a145a8b42123631b5d5a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.065651 4098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b681911790a145a8b42123631b5d5a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.068856 4327 tablet_bootstrap.cc:492] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Bootstrap starting.
I20260928 21:58:56.069504 4328 tablet_bootstrap.cc:492] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Bootstrap starting.
I20260928 21:58:56.069620 4327 tablet_bootstrap.cc:654] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.070036 4327 log.cc:824] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:56.070498 4328 tablet_bootstrap.cc:654] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.070856 4327 tablet_bootstrap.cc:492] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: No bootstrap required, opened a new log
I20260928 21:58:56.070838 4328 log.cc:824] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:56.070925 4327 ts_tablet_manager.cc:1397] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:56.071174 4329 tablet_bootstrap.cc:492] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Bootstrap starting.
I20260928 21:58:56.071528 4328 tablet_bootstrap.cc:492] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: No bootstrap required, opened a new log
I20260928 21:58:56.071612 4328 ts_tablet_manager.cc:1397] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:56.071820 4329 tablet_bootstrap.cc:654] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.072130 4329 log.cc:824] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:56.073208 4327 raft_consensus.cc:348] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.073315 4329 tablet_bootstrap.cc:492] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: No bootstrap required, opened a new log
I20260928 21:58:56.073369 4327 raft_consensus.cc:374] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.073388 4329 ts_tablet_manager.cc:1397] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:56.073412 4327 raft_consensus.cc:729] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabcce7f6b8c4cada79ae523c95840ef, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.073546 4327 consensus_queue.cc:260] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.073834 4327 ts_tablet_manager.cc:1428] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:56.073770 4328 raft_consensus.cc:348] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.073937 4328 raft_consensus.cc:374] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.073982 4328 raft_consensus.cc:729] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4c36c95d2d442f995e8f5646fe6d61, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.074029 4024 heartbeater.cc:503] Master 127.2.165.126:45689 was elected leader, sending a full tablet report...
I20260928 21:58:56.074093 4328 consensus_queue.cc:260] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.074353 4328 ts_tablet_manager.cc:1428] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:56.074389 4304 heartbeater.cc:503] Master 127.2.165.126:45689 was elected leader, sending a full tablet report...
I20260928 21:58:56.075464 4329 raft_consensus.cc:348] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.075587 4329 raft_consensus.cc:374] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.075608 4329 raft_consensus.cc:729] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d730df97f26428493b5ecb5db023f3d, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.075712 4329 consensus_queue.cc:260] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.075928 4329 ts_tablet_manager.cc:1428] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:56.075982 4164 heartbeater.cc:503] Master 127.2.165.126:45689 was elected leader, sending a full tablet report...
W20260928 21:58:56.101373 4165 tablet.cc:2369] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:58:56.193136 4025 tablet.cc:2369] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:56.245771 4334 raft_consensus.cc:482] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:56.245891 4334 raft_consensus.cc:504] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.246302 4334 leader_election.cc:288] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563), 3d730df97f26428493b5ecb5db023f3d (127.2.165.66:34809)
I20260928 21:58:56.249503 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.246586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39563 (local address 127.2.165.67:36103)
0928 21:58:56.246651 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.246664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.246709 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.246900 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.246901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.246903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:56.246996 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.246998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.247750 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.247751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.248595 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.248598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.249221 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.249222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.249317 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.249321 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.249373 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.249413 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":7}
I20260928 21:58:56.249516 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.246649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:36103 (local address 127.2.165.65:39563)
0928 21:58:56.246732 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.246734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.246747 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:58:56.246780 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.246780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.246791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.246866 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:56.247098 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.247654 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.248727 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.249144 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.249200 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.249207 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.249392 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.249437 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:58:56.249620 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.246825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:41191 (local address 127.2.165.66:34809)
0928 21:58:56.246876 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.246878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.246957 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:58:56.247023 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.247024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.247033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.247088 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:56.247297 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.247862 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.248841 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.249237 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.249312 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.249320 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.249497 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.249544 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:58:56.249634 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.246724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:34809 (local address 127.2.165.67:41191)
0928 21:58:56.246886 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.246897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.246957 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.247101 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.247102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.247103 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:56.247201 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.247202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.247950 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.247951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.248767 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.248769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.249323 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.249323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.249413 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.249417 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.249477 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.249515 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260928 21:58:56.250049 4118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b681911790a145a8b42123631b5d5a31" candidate_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730df97f26428493b5ecb5db023f3d" is_pre_election: true
I20260928 21:58:56.250061 3978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b681911790a145a8b42123631b5d5a31" candidate_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcce7f6b8c4cada79ae523c95840ef" is_pre_election: true
I20260928 21:58:56.250221 4118 raft_consensus.cc:2454] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e4c36c95d2d442f995e8f5646fe6d61 in term 0.
I20260928 21:58:56.250226 3978 raft_consensus.cc:2454] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e4c36c95d2d442f995e8f5646fe6d61 in term 0.
I20260928 21:58:56.250452 4191 leader_election.cc:302] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4c36c95d2d442f995e8f5646fe6d61, eabcce7f6b8c4cada79ae523c95840ef; no voters:
I20260928 21:58:56.250602 4334 raft_consensus.cc:2781] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:56.250707 4334 raft_consensus.cc:482] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:56.250727 4334 raft_consensus.cc:3037] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.251564 4334 raft_consensus.cc:504] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.251693 4334 leader_election.cc:288] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [CANDIDATE]: Term 1 election: Requested vote from peers eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563), 3d730df97f26428493b5ecb5db023f3d (127.2.165.66:34809)
I20260928 21:58:56.251892 3978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b681911790a145a8b42123631b5d5a31" candidate_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcce7f6b8c4cada79ae523c95840ef"
I20260928 21:58:56.251915 4118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b681911790a145a8b42123631b5d5a31" candidate_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d730df97f26428493b5ecb5db023f3d"
I20260928 21:58:56.251998 4118 raft_consensus.cc:3037] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.252002 3978 raft_consensus.cc:3037] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.252837 4118 raft_consensus.cc:2454] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4c36c95d2d442f995e8f5646fe6d61 in term 1.
I20260928 21:58:56.252875 3978 raft_consensus.cc:2454] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4c36c95d2d442f995e8f5646fe6d61 in term 1.
I20260928 21:58:56.253023 4191 leader_election.cc:302] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4c36c95d2d442f995e8f5646fe6d61, eabcce7f6b8c4cada79ae523c95840ef; no voters:
I20260928 21:58:56.253160 4334 raft_consensus.cc:2781] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:56.253360 4334 raft_consensus.cc:686] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 LEADER]: Becoming Leader. State: Replica: 1e4c36c95d2d442f995e8f5646fe6d61, State: Running, Role: LEADER
I20260928 21:58:56.253464 4334 consensus_queue.cc:237] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } }
I20260928 21:58:56.254480 3821 catalog_manager.cc:5495] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } health_report { overall_health: UNKNOWN } } }
W20260928 21:58:56.260886 4305 tablet.cc:2369] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:56.293824 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.291114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.2.165.67:33225)
0928 21:58:56.291164 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.291166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.291182 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:58:56.291264 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.291265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.291281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.291338 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:56.291469 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.291996 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.292909 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.293108 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.293165 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.293352 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:56.293449 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:56.293504 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.293595 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.293643 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:58:56.295719 4118 raft_consensus.cc:1261] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Refusing update from remote peer 1e4c36c95d2d442f995e8f5646fe6d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:56.295744 3978 raft_consensus.cc:1261] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Refusing update from remote peer 1e4c36c95d2d442f995e8f5646fe6d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:56.296053 4337 consensus_queue.cc:1035] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:56.296213 4334 consensus_queue.cc:1035] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:56.298796 4341 mvcc.cc:205] Tried to move back new op lower bound from 7334431687864356864 to 7334431687695003648. Current Snapshot: MvccSnapshot[applied={T|T < 7334431687864356864}]
I20260928 21:58:56.299595 4342 mvcc.cc:205] Tried to move back new op lower bound from 7334431687864356864 to 7334431687695003648. Current Snapshot: MvccSnapshot[applied={T|T < 7334431687864356864 or (T in {7334431687864356864})}]
I20260928 21:58:56.302169 3820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:56.302269 3820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:56.303206 3820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395: Sending DeleteTablet for 3 replicas of tablet b681911790a145a8b42123631b5d5a31
I20260928 21:58:56.303622 3958 tablet_service.cc:1514] Processing DeleteTablet for tablet b681911790a145a8b42123631b5d5a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:39068
I20260928 21:58:56.303647 4098 tablet_service.cc:1514] Processing DeleteTablet for tablet b681911790a145a8b42123631b5d5a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:34610
I20260928 21:58:56.303649 4238 tablet_service.cc:1514] Processing DeleteTablet for tablet b681911790a145a8b42123631b5d5a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:45798
I20260928 21:58:56.303870 4352 tablet_replica.cc:330] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: stopping tablet replica
I20260928 21:58:56.303894 3820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:58:56.303964 4352 raft_consensus.cc:2229] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:58:56.303963 4350 tablet_replica.cc:330] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: stopping tablet replica
W20260928 21:58:56.304029 3820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:56.304057 4350 raft_consensus.cc:2229] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:56.304054 4352 raft_consensus.cc:2258] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:56.304636 4352 ts_tablet_manager.cc:1905] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:56.305150 4351 tablet_replica.cc:330] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: stopping tablet replica
I20260928 21:58:56.305312 4351 raft_consensus.cc:2229] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:56.305498 4351 raft_consensus.cc:2258] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:56.305519 3958 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9bfa9b988c42ebb61a12674f293134 (DEFAULT_TABLE table=test-table [id=890f8d9b52a94a3d958475f3d6bd55f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:56.305661 3958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9bfa9b988c42ebb61a12674f293134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.306077 4351 ts_tablet_manager.cc:1905] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:56.306578 4352 ts_tablet_manager.cc:1918] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:56.306663 4352 log.cc:1196] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/b681911790a145a8b42123631b5d5a31
I20260928 21:58:56.306933 4352 ts_tablet_manager.cc:1939] T b681911790a145a8b42123631b5d5a31 P 1e4c36c95d2d442f995e8f5646fe6d61: Deleting consensus metadata
I20260928 21:58:56.307152 4327 tablet_bootstrap.cc:492] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: Bootstrap starting.
I20260928 21:58:56.307210 4098 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9bfa9b988c42ebb61a12674f293134 (DEFAULT_TABLE table=test-table [id=890f8d9b52a94a3d958475f3d6bd55f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:56.307319 4098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9bfa9b988c42ebb61a12674f293134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.307536 4327 tablet_bootstrap.cc:654] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.308099 3807 catalog_manager.cc:4841] TS 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67:33225): tablet b681911790a145a8b42123631b5d5a31 (table test-table [id=4bf59173ae114d31aee6a52519cc2b61]) successfully deleted
I20260928 21:58:56.308558 4327 tablet_bootstrap.cc:492] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: No bootstrap required, opened a new log
I20260928 21:58:56.308605 4327 ts_tablet_manager.cc:1397] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:56.308652 4238 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9bfa9b988c42ebb61a12674f293134 (DEFAULT_TABLE table=test-table [id=890f8d9b52a94a3d958475f3d6bd55f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:56.308779 4238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9bfa9b988c42ebb61a12674f293134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.308807 4327 raft_consensus.cc:348] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.308885 4327 raft_consensus.cc:374] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.308907 4327 raft_consensus.cc:729] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabcce7f6b8c4cada79ae523c95840ef, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.308938 4327 consensus_queue.cc:260] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.309043 4327 ts_tablet_manager.cc:1428] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:56.309166 4351 ts_tablet_manager.cc:1918] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:56.309278 4351 log.cc:1196] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/b681911790a145a8b42123631b5d5a31
I20260928 21:58:56.309589 4351 ts_tablet_manager.cc:1939] T b681911790a145a8b42123631b5d5a31 P 3d730df97f26428493b5ecb5db023f3d: Deleting consensus metadata
I20260928 21:58:56.310164 4328 tablet_bootstrap.cc:492] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: Bootstrap starting.
I20260928 21:58:56.310195 4329 tablet_bootstrap.cc:492] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: Bootstrap starting.
I20260928 21:58:56.310586 4329 tablet_bootstrap.cc:654] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.310585 4328 tablet_bootstrap.cc:654] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.311039 3807 catalog_manager.cc:4841] TS 3d730df97f26428493b5ecb5db023f3d (127.2.165.66:34809): tablet b681911790a145a8b42123631b5d5a31 (table test-table [id=4bf59173ae114d31aee6a52519cc2b61]) successfully deleted
I20260928 21:58:56.311435 4329 tablet_bootstrap.cc:492] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: No bootstrap required, opened a new log
I20260928 21:58:56.311482 4329 ts_tablet_manager.cc:1397] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:56.311642 4329 raft_consensus.cc:348] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.311722 4329 raft_consensus.cc:374] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.311750 4329 raft_consensus.cc:729] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d730df97f26428493b5ecb5db023f3d, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.311795 4329 consensus_queue.cc:260] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.311887 4329 ts_tablet_manager.cc:1428] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:56.311992 4350 raft_consensus.cc:2258] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:56.312455 4350 ts_tablet_manager.cc:1905] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:56.314549 4350 ts_tablet_manager.cc:1918] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:56.314641 4350 log.cc:1196] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/b681911790a145a8b42123631b5d5a31
I20260928 21:58:56.314698 4328 tablet_bootstrap.cc:492] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: No bootstrap required, opened a new log
I20260928 21:58:56.314752 4328 ts_tablet_manager.cc:1397] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 21:58:56.314900 4350 ts_tablet_manager.cc:1939] T b681911790a145a8b42123631b5d5a31 P eabcce7f6b8c4cada79ae523c95840ef: Deleting consensus metadata
I20260928 21:58:56.314914 4328 raft_consensus.cc:348] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.314991 4328 raft_consensus.cc:374] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.315018 4328 raft_consensus.cc:729] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4c36c95d2d442f995e8f5646fe6d61, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.315078 4328 consensus_queue.cc:260] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.315164 4328 ts_tablet_manager.cc:1428] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:56.315389 3807 catalog_manager.cc:4841] TS eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563): tablet b681911790a145a8b42123631b5d5a31 (table test-table [id=4bf59173ae114d31aee6a52519cc2b61]) successfully deleted
I20260928 21:58:56.518319 4335 raft_consensus.cc:482] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:56.518429 4335 raft_consensus.cc:504] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.518814 4335 leader_election.cc:288] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563), 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67:33225)
I20260928 21:58:56.522254 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.519395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:40507 (local address 127.2.165.67:33225)
0928 21:58:56.519441 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.519443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.519482 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:58:56.519572 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.519573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.519585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.519650 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:56.519838 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.520288 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.521472 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.521839 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.521906 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.521913 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.522084 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.522132 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:58:56.522257 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.519240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:55183 (local address 127.2.165.65:39563)
0928 21:58:56.519320 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.519323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.519356 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:58:56.519478 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.519479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.519494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.519556 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:56.519731 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.520288 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.521463 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.521892 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.521970 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.521980 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.522125 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.522167 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:58:56.522667 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.519096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39563 (local address 127.2.165.66:55183)
0928 21:58:56.519249 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.519264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.519355 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.519571 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.519572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.519575 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:56.519659 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.519662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.520376 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.520377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.521355 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.521358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.521974 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.521975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.522029 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.522032 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.522107 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.522600 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":94,"thread_start_us":75,"threads_started":1}
I20260928 21:58:56.522934 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.519319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:33225 (local address 127.2.165.66:40507)
0928 21:58:56.519417 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.519428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.519498 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.519674 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.519675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.519677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:56.519742 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.519745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.520407 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.520408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.521367 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.521369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.521920 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.521921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.521998 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.522003 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.522059 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.522851 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":26,"thread_start_us":65,"threads_started":1}
I20260928 21:58:56.522974 3978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9bfa9b988c42ebb61a12674f293134" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcce7f6b8c4cada79ae523c95840ef" is_pre_election: true
I20260928 21:58:56.523093 3978 raft_consensus.cc:2454] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 0.
I20260928 21:58:56.523191 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9bfa9b988c42ebb61a12674f293134" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" is_pre_election: true
I20260928 21:58:56.523306 4258 raft_consensus.cc:2454] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 0.
I20260928 21:58:56.523484 4051 leader_election.cc:302] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4c36c95d2d442f995e8f5646fe6d61, 3d730df97f26428493b5ecb5db023f3d; no voters:
I20260928 21:58:56.523623 4335 raft_consensus.cc:2781] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:56.523656 4335 raft_consensus.cc:482] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:56.523679 4335 raft_consensus.cc:3037] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.524343 4335 raft_consensus.cc:504] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.524480 4335 leader_election.cc:288] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 election: Requested vote from peers eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563), 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67:33225)
I20260928 21:58:56.524724 3978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9bfa9b988c42ebb61a12674f293134" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcce7f6b8c4cada79ae523c95840ef"
I20260928 21:58:56.524776 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9bfa9b988c42ebb61a12674f293134" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c36c95d2d442f995e8f5646fe6d61"
I20260928 21:58:56.524829 3978 raft_consensus.cc:3037] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.524860 4258 raft_consensus.cc:3037] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.525506 3978 raft_consensus.cc:2454] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 1.
I20260928 21:58:56.525521 4258 raft_consensus.cc:2454] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 1.
I20260928 21:58:56.525694 4051 leader_election.cc:302] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4c36c95d2d442f995e8f5646fe6d61, 3d730df97f26428493b5ecb5db023f3d; no voters:
I20260928 21:58:56.525812 4335 raft_consensus.cc:2781] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:56.525929 4335 raft_consensus.cc:686] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 1 LEADER]: Becoming Leader. State: Replica: 3d730df97f26428493b5ecb5db023f3d, State: Running, Role: LEADER
I20260928 21:58:56.526010 4335 consensus_queue.cc:237] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.526784 3820 catalog_manager.cc:5495] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d730df97f26428493b5ecb5db023f3d (127.2.165.66). New cstate: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[3776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:56.569344 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.561614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45689 (local address 127.0.0.1:36232)
0928 21:58:56.561942 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.561955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.562059 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.562240 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.562240 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.562615 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:56.562667 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.562669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.563351 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.563352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.563873 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.563874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.564225 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.564226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.564275 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.564965 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:56.564974 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:56.566320 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:56.568185 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:56.568193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:56.568196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:56.568441 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:56.568717 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:56.568719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:56.568719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:56.568795 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:56.569051 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:56.569057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:56.569106 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.569173 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.569268 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":189,"threads_started":1}
I20260928 21:58:56.569494 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.561724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.2.165.126:45689)
0928 21:58:56.561932 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.561934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.562036 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:58:56.562129 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.562129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.562141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.562233 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:56.562762 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.563271 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.563952 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.564144 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.564225 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.566411 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:56.566413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:56.566413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:56.566421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:56.568069 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.568517 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.568518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.568519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.568583 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.568856 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.568857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.568857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.568968 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:56.569021 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.569286 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.569357 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260928 21:58:56.571553 3820 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:36232:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:58:56.571766 3820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:56.573122 3958 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba7a26d3a144991b31b038c17657051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a05fa92adda4dd28310c5811dee04ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:58:56.573210 4098 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba7a26d3a144991b31b038c17657051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a05fa92adda4dd28310c5811dee04ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:58:56.573278 3958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba7a26d3a144991b31b038c17657051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.573374 4098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba7a26d3a144991b31b038c17657051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.574441 4327 tablet_bootstrap.cc:492] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: Bootstrap starting.
I20260928 21:58:56.574563 4329 tablet_bootstrap.cc:492] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: Bootstrap starting.
I20260928 21:58:56.574894 4327 tablet_bootstrap.cc:654] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.574991 4329 tablet_bootstrap.cc:654] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.575425 4238 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba7a26d3a144991b31b038c17657051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a05fa92adda4dd28310c5811dee04ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:58:56.575582 4238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba7a26d3a144991b31b038c17657051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:56.575608 4327 tablet_bootstrap.cc:492] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: No bootstrap required, opened a new log
I20260928 21:58:56.575652 4327 ts_tablet_manager.cc:1397] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:56.575613 4329 tablet_bootstrap.cc:492] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: No bootstrap required, opened a new log
I20260928 21:58:56.575711 4329 ts_tablet_manager.cc:1397] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:56.575819 4327 raft_consensus.cc:348] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.575899 4327 raft_consensus.cc:374] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.575861 4329 raft_consensus.cc:348] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.575922 4327 raft_consensus.cc:729] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabcce7f6b8c4cada79ae523c95840ef, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.575932 4329 raft_consensus.cc:374] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.575956 4329 raft_consensus.cc:729] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d730df97f26428493b5ecb5db023f3d, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.575974 4327 consensus_queue.cc:260] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.575995 4329 consensus_queue.cc:260] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.576083 4327 ts_tablet_manager.cc:1428] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:56.576078 4329 ts_tablet_manager.cc:1428] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:56.576140 4335 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.576223 4335 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:56.576275 4333 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.576352 4333 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:56.577066 4328 tablet_bootstrap.cc:492] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: Bootstrap starting.
I20260928 21:58:56.577443 4328 tablet_bootstrap.cc:654] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:56.578022 4328 tablet_bootstrap.cc:492] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: No bootstrap required, opened a new log
I20260928 21:58:56.578066 4328 ts_tablet_manager.cc:1397] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:56.578202 4328 raft_consensus.cc:348] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.578262 4328 raft_consensus.cc:374] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:56.578276 4328 raft_consensus.cc:729] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4c36c95d2d442f995e8f5646fe6d61, State: Initialized, Role: FOLLOWER
I20260928 21:58:56.578336 4328 consensus_queue.cc:260] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.578428 4328 ts_tablet_manager.cc:1428] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:56.578456 4345 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.578529 4345 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:56.723714 4335 raft_consensus.cc:482] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:56.723816 4335 raft_consensus.cc:504] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.723982 4335 leader_election.cc:288] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563), 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67:33225)
I20260928 21:58:56.724272 3978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7a26d3a144991b31b038c17657051" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcce7f6b8c4cada79ae523c95840ef" is_pre_election: true
I20260928 21:58:56.724362 3978 raft_consensus.cc:2454] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 0.
I20260928 21:58:56.724308 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7a26d3a144991b31b038c17657051" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" is_pre_election: true
I20260928 21:58:56.724447 4258 raft_consensus.cc:2454] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 0.
I20260928 21:58:56.724514 4051 leader_election.cc:302] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d730df97f26428493b5ecb5db023f3d, eabcce7f6b8c4cada79ae523c95840ef; no voters:
I20260928 21:58:56.724623 4335 raft_consensus.cc:2781] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:56.724695 4335 raft_consensus.cc:482] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:56.724730 4335 raft_consensus.cc:3037] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.725414 4335 raft_consensus.cc:504] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.725524 4335 leader_election.cc:288] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 election: Requested vote from peers eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563), 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67:33225)
I20260928 21:58:56.725797 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7a26d3a144991b31b038c17657051" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c36c95d2d442f995e8f5646fe6d61"
I20260928 21:58:56.725797 3978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba7a26d3a144991b31b038c17657051" candidate_uuid: "3d730df97f26428493b5ecb5db023f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabcce7f6b8c4cada79ae523c95840ef"
I20260928 21:58:56.725862 4258 raft_consensus.cc:3037] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.725886 3978 raft_consensus.cc:3037] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:56.726459 4258 raft_consensus.cc:2454] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 1.
I20260928 21:58:56.726537 3978 raft_consensus.cc:2454] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d730df97f26428493b5ecb5db023f3d in term 1.
I20260928 21:58:56.726672 4051 leader_election.cc:302] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4c36c95d2d442f995e8f5646fe6d61, 3d730df97f26428493b5ecb5db023f3d; no voters:
I20260928 21:58:56.726799 4335 raft_consensus.cc:2781] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:56.726886 4335 raft_consensus.cc:686] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [term 1 LEADER]: Becoming Leader. State: Replica: 3d730df97f26428493b5ecb5db023f3d, State: Running, Role: LEADER
I20260928 21:58:56.726938 4335 consensus_queue.cc:237] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } }
I20260928 21:58:56.727303 4358 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: TxnStatusTablet state changed. Reason: New leader 3d730df97f26428493b5ecb5db023f3d. Latest consensus state: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.727398 4358 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:58:56.727605 4371 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:58:56.727703 4371 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:58:56.727617 3820 catalog_manager.cc:5495] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d730df97f26428493b5ecb5db023f3d (127.2.165.66). New cstate: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } health_report { overall_health: UNKNOWN } } }
I20260928 21:58:56.760866 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.757733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:34809 (local address 127.0.0.1:34624)
0928 21:58:56.757839 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.757850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.757896 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.758179 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.758180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.758182 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:58:56.758240 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.758243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.759290 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.759292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.759882 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.759884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.760275 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.760276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.760333 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.760347 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:58:56.760636 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:58:56.760646 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.760739 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.760797 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 21:58:56.760867 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.757856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.2.165.66:34809)
0928 21:58:56.758026 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.758029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.758040 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:58:56.758072 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.758072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.758090 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.758174 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:56.758326 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.759188 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.759966 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.760191 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.760263 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.760459 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:56.760568 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:56.760654 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.760745 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.760767 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260928 21:58:56.762322 3978 raft_consensus.cc:1261] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Refusing update from remote peer 3d730df97f26428493b5ecb5db023f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:56.762355 4258 raft_consensus.cc:1261] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Refusing update from remote peer 3d730df97f26428493b5ecb5db023f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:56.762679 4335 consensus_queue.cc:1035] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:56.762820 4358 consensus_queue.cc:1035] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:56.763466 4333 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: TxnStatusTablet state changed. Reason: New leader 3d730df97f26428493b5ecb5db023f3d. Latest consensus state: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.763589 4333 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:56.763988 4345 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: TxnStatusTablet state changed. Reason: New leader 3d730df97f26428493b5ecb5db023f3d. Latest consensus state: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.764091 4345 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:56.764528 4358 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.764597 4335 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.764711 4335 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:58:56.764611 4358 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P 3d730df97f26428493b5ecb5db023f3d: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:58:56.765272 4340 mvcc.cc:205] Tried to move back new op lower bound from 7334431689775792128 to 7334431689633857536. Current Snapshot: MvccSnapshot[applied={T|T < 7334431689775792128}]
I20260928 21:58:56.765352 4345 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.765427 4345 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P 1e4c36c95d2d442f995e8f5646fe6d61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:56.765646 4333 tablet_replica.cc:439] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d730df97f26428493b5ecb5db023f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 } } peers { permanent_uuid: "3d730df97f26428493b5ecb5db023f3d" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34809 } } peers { permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 } } }
I20260928 21:58:56.765733 4333 tablet_replica.cc:442] T 2ba7a26d3a144991b31b038c17657051 P eabcce7f6b8c4cada79ae523c95840ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:56.768961 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.766445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.2.165.66:34809)
0928 21:58:56.766484 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.766486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.766547 (+ 61us) server_negotiation.cc:407] Connection header received
0928 21:58:56.766615 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.766616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.766627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.766676 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:56.766779 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.767417 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.768216 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.768450 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.768524 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.768705 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:56.768740 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:56.768788 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.768845 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.768883 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:56.774250 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.771441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:34809 (local address 127.2.165.66:37635)
0928 21:58:56.771603 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:56.771615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:56.771679 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:56.771796 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:56.771796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:56.771797 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:58:56.771851 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.771852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.772738 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.772739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.773334 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:56.773335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.773779 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.773780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:56.773834 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.773847 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:58:56.774044 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:58:56.774047 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:56.774104 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:58:56.774140 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260928 21:58:56.774255 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.771500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:37635 (local address 127.2.165.66:34809)
0928 21:58:56.771535 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.771537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.771679 (+ 142us) server_negotiation.cc:407] Connection header received
0928 21:58:56.771722 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.771723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.771732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.771794 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:56.771887 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.772602 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.773407 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.773701 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.773772 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.773924 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:56.773956 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:56.774007 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.774130 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.774159 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":16}
I20260928 21:58:56.776435 4258 raft_consensus.cc:1261] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Refusing update from remote peer 3d730df97f26428493b5ecb5db023f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:56.776600 3978 raft_consensus.cc:1261] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Refusing update from remote peer 3d730df97f26428493b5ecb5db023f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:56.776751 4377 consensus_queue.cc:1035] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4c36c95d2d442f995e8f5646fe6d61" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:56.776849 4358 consensus_queue.cc:1035] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eabcce7f6b8c4cada79ae523c95840ef" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:56.790602 3821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:56.790733 3821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:56.791354 3821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c13dd4fc0629460da948e1a29342b395: Sending DeleteTablet for 3 replicas of tablet 7f9bfa9b988c42ebb61a12674f293134
I20260928 21:58:56.791623 3958 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9bfa9b988c42ebb61a12674f293134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:39068
I20260928 21:58:56.791671 4238 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9bfa9b988c42ebb61a12674f293134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:45798
I20260928 21:58:56.791742 4350 tablet_replica.cc:330] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: stopping tablet replica
I20260928 21:58:56.791766 4352 tablet_replica.cc:330] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: stopping tablet replica
I20260928 21:58:56.791805 4350 raft_consensus.cc:2229] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:56.791817 4352 raft_consensus.cc:2229] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:56.791839 4350 raft_consensus.cc:2258] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:56.791851 4352 raft_consensus.cc:2258] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:56.791989 4097 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9bfa9b988c42ebb61a12674f293134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:34610
I20260928 21:58:56.792078 4351 tablet_replica.cc:330] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: stopping tablet replica
I20260928 21:58:56.792119 4351 raft_consensus.cc:2229] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:58:56.792124 4352 ts_tablet_manager.cc:1905] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:56.792203 4351 raft_consensus.cc:2258] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:56.792459 4350 ts_tablet_manager.cc:1905] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:56.793721 4351 ts_tablet_manager.cc:1905] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:56.793967 4352 ts_tablet_manager.cc:1918] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:58:56.794023 4352 log.cc:1196] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/7f9bfa9b988c42ebb61a12674f293134
I20260928 21:58:56.794019 4350 ts_tablet_manager.cc:1918] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:58:56.794075 4350 log.cc:1196] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/7f9bfa9b988c42ebb61a12674f293134
I20260928 21:58:56.794238 4352 ts_tablet_manager.cc:1939] T 7f9bfa9b988c42ebb61a12674f293134 P 1e4c36c95d2d442f995e8f5646fe6d61: Deleting consensus metadata
I20260928 21:58:56.794291 4350 ts_tablet_manager.cc:1939] T 7f9bfa9b988c42ebb61a12674f293134 P eabcce7f6b8c4cada79ae523c95840ef: Deleting consensus metadata
I20260928 21:58:56.794554 3807 catalog_manager.cc:4841] TS 1e4c36c95d2d442f995e8f5646fe6d61 (127.2.165.67:33225): tablet 7f9bfa9b988c42ebb61a12674f293134 (table test-table [id=890f8d9b52a94a3d958475f3d6bd55f7]) successfully deleted
I20260928 21:58:56.795140 3807 catalog_manager.cc:4841] TS eabcce7f6b8c4cada79ae523c95840ef (127.2.165.65:39563): tablet 7f9bfa9b988c42ebb61a12674f293134 (table test-table [id=890f8d9b52a94a3d958475f3d6bd55f7]) successfully deleted
I20260928 21:58:56.795567 4351 ts_tablet_manager.cc:1918] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:58:56.795648 4351 log.cc:1196] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/7f9bfa9b988c42ebb61a12674f293134
I20260928 21:58:56.795881 4351 ts_tablet_manager.cc:1939] T 7f9bfa9b988c42ebb61a12674f293134 P 3d730df97f26428493b5ecb5db023f3d: Deleting consensus metadata
I20260928 21:58:56.796365 3807 catalog_manager.cc:4841] TS 3d730df97f26428493b5ecb5db023f3d (127.2.165.66:34809): tablet 7f9bfa9b988c42ebb61a12674f293134 (table test-table [id=890f8d9b52a94a3d958475f3d6bd55f7]) successfully deleted
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[3776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632736, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:58:56.805326 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.795277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.2.165.65:39563)
0928 21:58:56.795320 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.795323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.795635 (+ 312us) server_negotiation.cc:407] Connection header received
0928 21:58:56.795720 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.795721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.795732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.795776 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:56.796574 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.797236 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.798253 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.798432 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.798480 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.801961 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:56.801975 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:56.801976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:56.801985 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:56.803982 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.804421 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.804425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.804426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.804489 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.804724 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.804725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.804725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.804881 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:56.805046 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.805120 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.805211 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:58:56.805706 4018 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072
I20260928 21:58:56.813392 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:56.806831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.2.165.126:45689)
0928 21:58:56.806885 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:56.806888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:56.806897 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:58:56.806974 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:56.806975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:56.806987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:56.807043 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:56.807566 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.808100 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.808859 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:56.809108 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:56.809181 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:56.810550 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:56.810551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:56.810551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:56.810559 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:56.811997 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.812403 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.812403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.812404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.812461 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:56.812776 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:56.812777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:56.812778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:56.812901 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:56.813042 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:56.813163 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 21:58:56.813281 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 21:58:56.813663 3821 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36240
I20260928 21:58:56.814764 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 3893
I20260928 21:58:56.822614 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 4033
I20260928 21:58:56.828800 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 4173
I20260928 21:58:56.834697 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 3789
2026-09-28T21:58:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2676 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[4390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[4390](info): set up 2 sockets
Sep 28 21:58:56 dist-test-slave-3t36 krb5kdc[4390](info): commencing operation
krb5kdc: starting...
W20260928 21:58:57.908447 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:58:57 dist-test-slave-3t36 krb5kdc[4390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:58:58.031163 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:40139
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:40139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:58.114158 4403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:58.114395 4403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:58.114418 4403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:58.115531 4403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:58.115567 4403 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:58:58.115586 4403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:58.115600 4403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:58.115612 4403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:58.115624 4403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:58.117631 4403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:40139
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:58.117925 4403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:58.118216 4403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:58:58.123526 4403 server_base.cc:1034] running on GCE node
W20260928 21:58:58.123486 4408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:58.123462 4411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:58.123466 4409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:58.124169 4403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:58.124533 4403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:58.125808 4403 hybrid_clock.cc:648] HybridClock initialized: now 1790632738125768 us; error 54 us; skew 500 ppm
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:58.128491 4403 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:58:58.129066 4403 webserver.cc:460] Webserver started at http://127.2.165.126:35141/ using document root <none> and password file <none>
I20260928 21:58:58.129300 4403 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:58.129365 4403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:58.129516 4403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:58.130620 4403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "9f4547de5b674602a2e1550f53655554"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "4e2373642ccadd7fb208cf2e005e9e6e"
server_key_iv: "2fe8af1b5ccedec3a6a95d30362a6268"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.131008 4403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "9f4547de5b674602a2e1550f53655554"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "4e2373642ccadd7fb208cf2e005e9e6e"
server_key_iv: "2fe8af1b5ccedec3a6a95d30362a6268"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.132956 4403 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 21:58:58.133912 4418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.134259 4403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:58.134372 4403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "9f4547de5b674602a2e1550f53655554"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "4e2373642ccadd7fb208cf2e005e9e6e"
server_key_iv: "2fe8af1b5ccedec3a6a95d30362a6268"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.134465 4403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:58.146241 4403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:58.148437 4403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:58.148623 4403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:58.154143 4403 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:40139
I20260928 21:58:58.154232 4480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:40139 every 8 connection(s)
I20260928 21:58:58.154891 4403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:58.155814 4481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:58.156405 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 4403
I20260928 21:58:58.156481 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:58:58.156661 2709 external_mini_cluster.cc:1428] Setting key 6409594e06e0f7559822e5042a74b444
I20260928 21:58:58.158921 4481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Bootstrap starting.
I20260928 21:58:58.159556 4481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:58.159866 4481 log.cc:824] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:58.160584 4481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: No bootstrap required, opened a new log
I20260928 21:58:58.162815 4481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4547de5b674602a2e1550f53655554" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40139 } }
I20260928 21:58:58.163000 4481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:58.163054 4481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4547de5b674602a2e1550f53655554, State: Initialized, Role: FOLLOWER
I20260928 21:58:58.163230 4481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4547de5b674602a2e1550f53655554" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40139 } }
I20260928 21:58:58.163335 4481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:58.163383 4481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:58.163435 4481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:58.164120 4481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4547de5b674602a2e1550f53655554" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40139 } }
I20260928 21:58:58.164249 4481 leader_election.cc:302] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f4547de5b674602a2e1550f53655554; no voters:
I20260928 21:58:58.164480 4481 leader_election.cc:288] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:58.164521 4486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:58.164669 4486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [term 1 LEADER]: Becoming Leader. State: Replica: 9f4547de5b674602a2e1550f53655554, State: Running, Role: LEADER
I20260928 21:58:58.164850 4486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4547de5b674602a2e1550f53655554" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40139 } }
I20260928 21:58:58.164886 4481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:58.165548 4487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f4547de5b674602a2e1550f53655554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4547de5b674602a2e1550f53655554" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40139 } } }
I20260928 21:58:58.165640 4487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:58.165874 4495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:58.166389 4495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:58.166848 4488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f4547de5b674602a2e1550f53655554. Latest consensus state: current_term: 1 leader_uuid: "9f4547de5b674602a2e1550f53655554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4547de5b674602a2e1550f53655554" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40139 } } }
I20260928 21:58:58.166934 4488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:58.168181 4495 catalog_manager.cc:1347] Generated new cluster ID: 8f7ced3419e54621a10cf0b9e8d674a0
I20260928 21:58:58.168236 4495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:58.168884 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.157303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.2.165.126:40139)
0928 21:58:58.157479 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.157487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.157514 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:58:58.157956 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.157963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.158015 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.158147 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.158936 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.160025 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.160963 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.161428 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.161564 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.164221 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.164233 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.164233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.164246 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.166583 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.167098 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.167102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.167102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.167218 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.167423 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.167424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.167424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.167848 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.168411 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.168471 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.168645 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260928 21:58:58.210189 4495 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:58.210772 4495 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:58.216207 4495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Generated new TSK 0
I20260928 21:58:58.216408 4495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:58:58.253458 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40139
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:58.339202 4507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:58.339440 4507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:58.339470 4507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:58.340739 4507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:58.340782 4507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:58.340806 4507 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:58:58.340817 4507 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:58:58.340835 4507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:58:58.342396 4507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:58.342696 4507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:58.342949 4507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:58.348615 4515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:58.348615 4512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:58.348726 4513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:58.348855 4507 server_base.cc:1034] running on GCE node
I20260928 21:58:58.349247 4507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:58.349478 4507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:58.350641 4507 hybrid_clock.cc:648] HybridClock initialized: now 1790632738350629 us; error 34 us; skew 500 ppm
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:58.353384 4507 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:58:58.353863 4507 webserver.cc:460] Webserver started at http://127.2.165.65:43853/ using document root <none> and password file <none>
I20260928 21:58:58.354112 4507 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:58.354182 4507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:58.354344 4507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:58.355419 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "8c6dc716658f45b0914bf55849910fdb"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "ec91e0f083a011eab3e881f83da8617d"
server_key_iv: "37d7adeb2207feb83bf72aed19e115e2"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.355792 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "8c6dc716658f45b0914bf55849910fdb"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "ec91e0f083a011eab3e881f83da8617d"
server_key_iv: "37d7adeb2207feb83bf72aed19e115e2"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.357311 4507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:58.358131 4522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.358356 4507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:58.358466 4507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "8c6dc716658f45b0914bf55849910fdb"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "ec91e0f083a011eab3e881f83da8617d"
server_key_iv: "37d7adeb2207feb83bf72aed19e115e2"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.358544 4507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:58.378461 4507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:58.381170 4507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:58.381335 4507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:58.381894 4507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:58.381934 4507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.381958 4507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:58.381974 4507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.389766 4507 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:38345
I20260928 21:58:58.390019 4637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:38345 every 8 connection(s)
I20260928 21:58:58.390208 4507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:58.394344 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.382508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:34645 (local address 127.2.165.126:40139)
0928 21:58:58.382587 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.382589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.382848 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:58:58.383727 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.383730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.383744 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.383827 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.384915 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.385809 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.387551 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.387771 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.387842 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.391287 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.391289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.391290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.391299 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.392793 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.393246 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.393248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.393248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.393304 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.393604 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.393605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.393606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.393792 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.393858 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.394183 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.394279 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:58:58.394424 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.382467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40139 (local address 127.2.165.65:34645)
0928 21:58:58.382706 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.382750 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.383539 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.383936 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.383941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.384350 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:58.384687 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.384695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.385936 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.385937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.387439 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.387442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.387888 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.387889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.388076 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.389055 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:58.389067 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:58.391150 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:58.392891 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.392897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.392899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.393165 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.393373 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.393374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.393374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.393482 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.393874 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:58.393880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:58.394013 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.394108 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.394256 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":162,"threads_started":1}
I20260928 21:58:58.395049 4638 heartbeater.cc:347] Connected to a master server at 127.2.165.126:40139
I20260928 21:58:58.395176 4638 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:58.395390 4638 heartbeater.cc:511] Master 127.2.165.126:40139 requested a full tablet report, sending...
I20260928 21:58:58.395903 4434 ts_manager.cc:194] Registered new tserver with Master: 8c6dc716658f45b0914bf55849910fdb (127.2.165.65:38345)
I20260928 21:58:58.396795 4434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:34645
I20260928 21:58:58.399099 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 4507
I20260928 21:58:58.399348 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:58.399575 2709 external_mini_cluster.cc:1428] Setting key c6bbcadaa98a3bc099c2abd217824b57
I20260928 21:58:58.402972 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.396189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40139 (local address 127.2.165.65:35473)
0928 21:58:58.396348 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.396365 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.396417 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.396538 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.396539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.396750 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:58.396818 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.396820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.397615 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.397616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.398138 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.398140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.398610 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.398612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.398762 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.399275 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:58.399281 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:58.399765 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:58.401789 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.401790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.401791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.401984 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.402251 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.402252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.402252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.402321 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.402610 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:58.402611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:58.402670 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.402758 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.402873 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260928 21:58:58.402966 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.396249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:35473 (local address 127.2.165.126:40139)
0928 21:58:58.396293 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.396296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.396422 (+ 126us) server_negotiation.cc:407] Connection header received
0928 21:58:58.396451 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.396452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.396462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.396526 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.396890 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.397485 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.398250 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.398451 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.398546 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.399911 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.399912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.399912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.399920 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.401703 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.402076 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.402078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.402078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.402173 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.402406 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.402407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.402407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.402526 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.402615 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.402783 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.402871 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:58:58.428954 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40139
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:58.514185 4647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:58.514421 4647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:58.514449 4647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:58.515614 4647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:58.515662 4647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:58.515686 4647 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:58:58.515702 4647 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:58:58.515719 4647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:58:58.517225 4647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:58.517483 4647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:58.517752 4647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:58.523033 4655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:58.523077 4647 server_base.cc:1034] running on GCE node
W20260928 21:58:58.523057 4653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:58.523057 4652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:58.523522 4647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:58.523885 4647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:58.525036 4647 hybrid_clock.cc:648] HybridClock initialized: now 1790632738525023 us; error 38 us; skew 500 ppm
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:58.527941 4647 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:58:58.528625 4647 webserver.cc:460] Webserver started at http://127.2.165.66:37869/ using document root <none> and password file <none>
I20260928 21:58:58.528915 4647 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:58.528968 4647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:58.529067 4647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:58.530157 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "5750dc33100a461e8703a63da45a60f3"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "2d17181fddb49bf15732f004077ddda1"
server_key_iv: "3ee54befb10871c8d8791da775c536f1"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.530489 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "5750dc33100a461e8703a63da45a60f3"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "2d17181fddb49bf15732f004077ddda1"
server_key_iv: "3ee54befb10871c8d8791da775c536f1"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.531876 4647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:58.532879 4662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.533144 4647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:58.533252 4647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "5750dc33100a461e8703a63da45a60f3"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "2d17181fddb49bf15732f004077ddda1"
server_key_iv: "3ee54befb10871c8d8791da775c536f1"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.533321 4647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:58.540089 4647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:58.542421 4647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:58.542563 4647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:58.543136 4647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:58.543181 4647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.543207 4647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:58.543220 4647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:58.550900 4647 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:42977
I20260928 21:58:58.550967 4777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:42977 every 8 connection(s)
I20260928 21:58:58.551290 4647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:58.553791 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.543744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:47521 (local address 127.2.165.126:40139)
0928 21:58:58.543805 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.543812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.544237 (+ 425us) server_negotiation.cc:407] Connection header received
0928 21:58:58.545083 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.545084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.545098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.545151 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.545844 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.546450 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.547460 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.547670 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.547752 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.550862 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.550865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.550865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.550876 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.552431 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.552955 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.552957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.552957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.553025 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.553254 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.553255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.553255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.553363 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.553412 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.553654 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.553725 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:58:58.554039 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.543740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40139 (local address 127.2.165.66:47521)
0928 21:58:58.544075 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.544128 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.544965 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.545222 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.545226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.545542 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:58.545745 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.545750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.546536 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.546537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.547367 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.547369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.547777 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.547778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.547930 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.548606 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:58.548620 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:58.550704 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:58.552514 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.552520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.552522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.552864 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.553102 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.553102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.553103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.553212 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.553415 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:58.553419 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:58.553552 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.553632 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.553811 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260928 21:58:58.554330 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 4647
I20260928 21:58:58.554431 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:58.554607 2709 external_mini_cluster.cc:1428] Setting key 073d3235f79eb1db7d18da2e2d57f78b
I20260928 21:58:58.555264 4778 heartbeater.cc:347] Connected to a master server at 127.2.165.126:40139
I20260928 21:58:58.555364 4778 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:58.555555 4778 heartbeater.cc:511] Master 127.2.165.126:40139 requested a full tablet report, sending...
I20260928 21:58:58.557165 4434 ts_manager.cc:194] Registered new tserver with Master: 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977)
I20260928 21:58:58.557651 4434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:47521
I20260928 21:58:58.563686 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.556338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40139 (local address 127.2.165.66:56765)
0928 21:58:58.556507 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.556519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.556583 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.557050 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.557051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.557303 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:58.557378 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.557381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.558342 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.558344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.559280 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.559283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.559793 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.559794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.559885 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.560442 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:58.560446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:58.560887 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:58.562484 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.562485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.562486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.562717 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.562949 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.562949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.562950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.563017 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.563333 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:58.563334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:58.563386 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.563466 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.563583 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260928 21:58:58.563681 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.556410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:56765 (local address 127.2.165.126:40139)
0928 21:58:58.556476 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.556479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.556575 (+ 96us) server_negotiation.cc:407] Connection header received
0928 21:58:58.556640 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.556641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.556649 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.556832 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.557462 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.558228 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.559394 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.559697 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.559760 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.560980 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.560981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.560982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.560989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.562386 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.562808 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.562810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.562810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.562876 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.563081 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.563082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.563083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.563246 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.563331 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.563502 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.563581 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:58:58.584121 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40139
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:58.669939 4787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:58.670167 4787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:58.670195 4787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:58.671352 4787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:58.671393 4787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:58.671414 4787 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:58:58.671427 4787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:58:58.671444 4787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:58:58.672998 4787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:58.673219 4787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:58.673412 4787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:58.678784 4792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:58.678992 4793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:58.679092 4795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:58.679040 4787 server_base.cc:1034] running on GCE node
I20260928 21:58:58.679351 4787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:58.679584 4787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:58.680807 4787 hybrid_clock.cc:648] HybridClock initialized: now 1790632738680834 us; error 75 us; skew 500 ppm
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:58.683575 4787 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:58:58.684104 4787 webserver.cc:460] Webserver started at http://127.2.165.67:38505/ using document root <none> and password file <none>
I20260928 21:58:58.684346 4787 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:58.684415 4787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:58.684573 4787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:58.685765 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "931383f03b0c4c62a3a3599ecb1d922c"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "e28de35eb84cf33c427bae1d2f9589e7"
server_key_iv: "9caa9680e9331f03db71d7f9ca0d8489"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.686208 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "931383f03b0c4c62a3a3599ecb1d922c"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "e28de35eb84cf33c427bae1d2f9589e7"
server_key_iv: "9caa9680e9331f03db71d7f9ca0d8489"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.687639 4787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:58.688640 4802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.689028 4787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:58.689162 4787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "931383f03b0c4c62a3a3599ecb1d922c"
format_stamp: "Formatted at 2026-09-28 21:58:58 on dist-test-slave-3t36"
server_key: "e28de35eb84cf33c427bae1d2f9589e7"
server_key_iv: "9caa9680e9331f03db71d7f9ca0d8489"
server_key_version: "encryptionkey@0"
I20260928 21:58:58.689265 4787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:58.710122 4787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:58.712606 4787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:58.712806 4787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:58.713475 4787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:58.713526 4787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.713560 4787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:58.713583 4787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:58.720981 4787 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:37939
I20260928 21:58:58.721087 4917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:37939 every 8 connection(s)
I20260928 21:58:58.721457 4787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:58.723681 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.713912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:39297 (local address 127.2.165.126:40139)
0928 21:58:58.713997 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.714000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.714235 (+ 235us) server_negotiation.cc:407] Connection header received
0928 21:58:58.714811 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.714812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.714825 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.714883 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.715845 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.716767 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.717919 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.718129 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.718229 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.720528 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.720530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.720530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.720539 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.722229 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.722673 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.722675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.722675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.722786 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.723057 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.723058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.723059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.723177 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.723239 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.723533 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.723603 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:58:58.723876 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.713906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40139 (local address 127.2.165.67:39297)
0928 21:58:58.714090 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.714130 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.714690 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.714982 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.714987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.715423 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:58.715727 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.715734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.716899 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.716900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.717820 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.717822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.718251 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.718252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.718390 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.718946 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:58.718954 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:58.720417 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:58.722311 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.722316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.722318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.722544 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.722873 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.722874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.722875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.722992 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.723261 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:58.723267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:58.723397 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.723499 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.723658 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":110,"threads_started":1}
I20260928 21:58:58.724351 4918 heartbeater.cc:347] Connected to a master server at 127.2.165.126:40139
I20260928 21:58:58.724447 4918 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:58.724639 4918 heartbeater.cc:511] Master 127.2.165.126:40139 requested a full tablet report, sending...
I20260928 21:58:58.725063 4434 ts_manager.cc:194] Registered new tserver with Master: 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:58:58.725512 4434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:39297
I20260928 21:58:58.729811 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 4787
I20260928 21:58:58.729904 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:58.730098 2709 external_mini_cluster.cc:1428] Setting key c8a7c9749266d9166851843705bfa3cd
I20260928 21:58:58.731046 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:58.731621 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.725818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:46923 (local address 127.2.165.126:40139)
0928 21:58:58.725909 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.725911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.725986 (+ 75us) server_negotiation.cc:407] Connection header received
0928 21:58:58.726056 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.726056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.726064 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.726110 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.726485 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.726990 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.727683 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.727897 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.727963 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.728916 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.728917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.728917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.728923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.730324 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.730765 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.730766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.730767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.730830 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.731054 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.731055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.731056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.731197 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.731269 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.731461 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.731531 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:58:58.731632 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.725785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40139 (local address 127.2.165.67:46923)
0928 21:58:58.725927 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.725939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.726001 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.726116 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.726116 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.726347 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:58.726412 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.726414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.727081 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.727082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.727596 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.727597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.727964 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.727964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.728018 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.728435 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:58.728439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:58.728840 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:58.730414 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.730415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.730416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.730616 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.730917 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.730917 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.730918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.730987 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.731269 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:58.731270 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:58.731315 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.731408 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.731528 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:58.755086 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.747876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.2.165.126:40139)
0928 21:58:58.747942 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.747944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.747957 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:58:58.748000 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.748010 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.748019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.748081 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.748702 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.749207 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.749972 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.750167 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.750252 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.752530 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.752531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.752531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.752538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.753978 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.754334 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.754335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.754335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.754392 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.754573 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.754574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.754575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.754690 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.754774 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.754924 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.754994 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:58:58.756995 4434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:58.758626 4434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:58.765511 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.761030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:37939 (local address 127.0.0.1:44692)
0928 21:58:58.761239 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.761273 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.761344 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.761696 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.761699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.761702 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:58.761809 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.761813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.763124 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.763125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.764210 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.764213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.765213 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.765214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.765265 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.765268 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.765329 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.765402 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":8,"thread_start_us":92,"threads_started":1}
I20260928 21:58:58.765722 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.761382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.2.165.67:37939)
0928 21:58:58.761533 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.761537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.761545 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:58.761577 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.761579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.761591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.761652 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:58.761899 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.763041 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.764299 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.765022 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.765122 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.765188 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.765573 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.765629 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 21:58:58.765851 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.761220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.2.165.65:38345)
0928 21:58:58.762006 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.762011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.762019 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:58.762055 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.762057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.762071 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.762145 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:58.762309 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.763310 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.764249 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.765018 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.765093 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.765155 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.765356 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.765401 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":80,"threads_started":1}
I20260928 21:58:58.765990 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.761224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.2.165.66:42977)
0928 21:58:58.761541 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.761545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.761554 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:58:58.761585 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.761587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.761599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.761704 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:58.761886 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.762829 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.764541 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.765297 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.765388 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.765439 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.765839 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.765887 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":72,"threads_started":1}
I20260928 21:58:58.765511 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.761031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38345 (local address 127.0.0.1:35228)
0928 21:58:58.761789 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.761799 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.761842 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.762182 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.762183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.762185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:58.762230 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.762232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.763396 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.763396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.764144 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.764146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.765122 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.765123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.765209 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.765225 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.765329 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.765401 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":125,"thread_start_us":68,"threads_started":1}
I20260928 21:58:58.765769 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.761029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42977 (local address 127.0.0.1:43186)
0928 21:58:58.761230 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.761273 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.761344 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.761712 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.761714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.761716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:58.761796 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.761798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.762914 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.762916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.764439 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.764441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.765477 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.765478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.765573 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.765580 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.765636 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.765679 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260928 21:58:58.766731 4852 tablet_service.cc:1467] Processing CreateTablet for tablet a679645c9d1c47eb9491805424fd6416 (DEFAULT_TABLE table=test-table [id=7c661e4369d84ef2a2f89ae33602c2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:58.767055 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679645c9d1c47eb9491805424fd6416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:58.769902 4941 tablet_bootstrap.cc:492] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Bootstrap starting.
I20260928 21:58:58.770426 4572 tablet_service.cc:1467] Processing CreateTablet for tablet a679645c9d1c47eb9491805424fd6416 (DEFAULT_TABLE table=test-table [id=7c661e4369d84ef2a2f89ae33602c2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:58.770754 4572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679645c9d1c47eb9491805424fd6416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:58.770958 4941 tablet_bootstrap.cc:654] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:58.771258 4941 log.cc:824] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:58.771960 4941 tablet_bootstrap.cc:492] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: No bootstrap required, opened a new log
I20260928 21:58:58.772017 4941 ts_tablet_manager.cc:1397] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:58.773710 4941 raft_consensus.cc:348] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.773869 4941 raft_consensus.cc:374] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:58.773912 4941 raft_consensus.cc:729] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931383f03b0c4c62a3a3599ecb1d922c, State: Initialized, Role: FOLLOWER
I20260928 21:58:58.773904 4943 tablet_bootstrap.cc:492] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Bootstrap starting.
I20260928 21:58:58.774102 4941 consensus_queue.cc:260] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.774395 4941 ts_tablet_manager.cc:1428] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:58:58.774421 4918 heartbeater.cc:503] Master 127.2.165.126:40139 was elected leader, sending a full tablet report...
I20260928 21:58:58.774536 4943 tablet_bootstrap.cc:654] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:58.774622 4712 tablet_service.cc:1467] Processing CreateTablet for tablet a679645c9d1c47eb9491805424fd6416 (DEFAULT_TABLE table=test-table [id=7c661e4369d84ef2a2f89ae33602c2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:58.774861 4943 log.cc:824] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:58.775009 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679645c9d1c47eb9491805424fd6416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:58.775748 4943 tablet_bootstrap.cc:492] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: No bootstrap required, opened a new log
I20260928 21:58:58.775810 4943 ts_tablet_manager.cc:1397] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:58.777428 4943 raft_consensus.cc:348] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.777612 4943 raft_consensus.cc:374] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:58.777639 4943 raft_consensus.cc:729] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Initialized, Role: FOLLOWER
I20260928 21:58:58.777774 4943 consensus_queue.cc:260] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.778048 4943 ts_tablet_manager.cc:1428] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:58:58.778092 4638 heartbeater.cc:503] Master 127.2.165.126:40139 was elected leader, sending a full tablet report...
I20260928 21:58:58.778474 4947 tablet_bootstrap.cc:492] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Bootstrap starting.
I20260928 21:58:58.779224 4947 tablet_bootstrap.cc:654] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:58.779470 4947 log.cc:824] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:58.780144 4947 tablet_bootstrap.cc:492] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: No bootstrap required, opened a new log
I20260928 21:58:58.780200 4947 ts_tablet_manager.cc:1397] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:58.781597 4947 raft_consensus.cc:348] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.781733 4947 raft_consensus.cc:374] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:58.781755 4947 raft_consensus.cc:729] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5750dc33100a461e8703a63da45a60f3, State: Initialized, Role: FOLLOWER
I20260928 21:58:58.781865 4947 consensus_queue.cc:260] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.782089 4947 ts_tablet_manager.cc:1428] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:58.782153 4778 heartbeater.cc:503] Master 127.2.165.126:40139 was elected leader, sending a full tablet report...
W20260928 21:58:58.801923 4779 tablet.cc:2369] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:58.813151 4946 raft_consensus.cc:482] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:58.813295 4946 raft_consensus.cc:504] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.813658 4946 leader_election.cc:288] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:58:58.817852 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.813949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:59839 (local address 127.2.165.66:42977)
0928 21:58:58.814016 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.814018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.814346 (+ 328us) server_negotiation.cc:407] Connection header received
0928 21:58:58.814376 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.814377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.814384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.814429 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:58.814560 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.815281 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.816854 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.817401 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.817485 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.817494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.817698 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.817784 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:58:58.817847 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.813862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:37939 (local address 127.2.165.65:58809)
0928 21:58:58.813906 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.813918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.813987 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.814157 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.814158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.814160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:58.814261 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.814264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.815355 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.815355 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.816757 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.816760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.817512 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.817513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.817700 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.817703 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.817753 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.817784 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":91}
I20260928 21:58:58.817880 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.813950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:58809 (local address 127.2.165.67:37939)
0928 21:58:58.814025 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.814027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.814037 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:58.814066 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.814067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.814077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.814133 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:58.814472 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.815247 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.816854 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.817420 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.817485 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.817494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.817778 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.817817 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:58:58.817858 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.813862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42977 (local address 127.2.165.65:59839)
0928 21:58:58.814253 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.814264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.814330 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.814444 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.814445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.814446 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:58.814493 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.814494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.815354 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.815355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.816757 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.816760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.817512 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.817513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.817621 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.817626 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.817692 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.817743 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":279,"thread_start_us":136,"threads_started":1}
I20260928 21:58:58.818352 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679645c9d1c47eb9491805424fd6416" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c" is_pre_election: true
I20260928 21:58:58.818426 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679645c9d1c47eb9491805424fd6416" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3" is_pre_election: true
I20260928 21:58:58.818518 4872 raft_consensus.cc:2454] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:58:58.818609 4732 raft_consensus.cc:2454] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:58:58.818817 4526 leader_election.cc:302] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c6dc716658f45b0914bf55849910fdb, 931383f03b0c4c62a3a3599ecb1d922c; no voters:
I20260928 21:58:58.819012 4946 raft_consensus.cc:2781] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:58.819110 4946 raft_consensus.cc:482] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:58.819149 4946 raft_consensus.cc:3037] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:58.819950 4946 raft_consensus.cc:504] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.820161 4946 leader_election.cc:288] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Requested vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:58:58.820317 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679645c9d1c47eb9491805424fd6416" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3"
I20260928 21:58:58.820325 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679645c9d1c47eb9491805424fd6416" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c"
I20260928 21:58:58.820425 4872 raft_consensus.cc:3037] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:58.820402 4732 raft_consensus.cc:3037] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:58.821244 4872 raft_consensus.cc:2454] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:58:58.821365 4732 raft_consensus.cc:2454] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:58:58.821411 4526 leader_election.cc:302] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c6dc716658f45b0914bf55849910fdb, 931383f03b0c4c62a3a3599ecb1d922c; no voters:
I20260928 21:58:58.821503 4946 raft_consensus.cc:2781] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:58.821681 4946 raft_consensus.cc:686] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 1 LEADER]: Becoming Leader. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Running, Role: LEADER
I20260928 21:58:58.821797 4946 consensus_queue.cc:237] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } }
I20260928 21:58:58.822678 4433 catalog_manager.cc:5495] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c6dc716658f45b0914bf55849910fdb (127.2.165.65). New cstate: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } health_report { overall_health: HEALTHY } } }
Sep 28 21:58:58 dist-test-slave-3t36 krb5kdc[4390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:58:58.865116 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.857532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.2.165.126:40139)
0928 21:58:58.857607 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:58.857610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:58.857777 (+ 167us) server_negotiation.cc:407] Connection header received
0928 21:58:58.857804 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:58.857804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:58.857814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:58.857878 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:58.858393 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.858889 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.859628 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.859846 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.859905 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.862508 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:58.862510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:58.862510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:58.862522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:58.863796 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.864141 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.864142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.864142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.864189 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:58.864395 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:58.864395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:58.864396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:58.864501 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:58.864553 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:58.864927 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 21:58:58.864998 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:58:58.865229 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:58.857436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40139 (local address 127.0.0.1:50722)
0928 21:58:58.857674 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:58.857685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:58.857753 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:58.857924 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:58.857925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:58.858263 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:58.858313 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.858315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.858986 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.858987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.859539 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:58.859541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:58.859930 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:58.859931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:58.859973 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:58.860804 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:58.860810 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:58.862426 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:58.863874 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.863881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.863883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.864077 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.864249 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:58.864250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:58.864250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:58.864310 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:58.864566 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:58.864572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:58.864616 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:58:58.864914 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 21:58:58.865104 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":39,"thread_start_us":87,"threads_started":1}
I20260928 21:58:58.867488 4433 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:50722:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:58:58.867795 4433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:58.869434 4572 tablet_service.cc:1467] Processing CreateTablet for tablet 5967eceb62a3451c82621daebdfbcddc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16221ac2b88945c29469c7f9370427a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:58:58.869488 4712 tablet_service.cc:1467] Processing CreateTablet for tablet 5967eceb62a3451c82621daebdfbcddc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16221ac2b88945c29469c7f9370427a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:58:58.869632 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5967eceb62a3451c82621daebdfbcddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:58.869681 4572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5967eceb62a3451c82621daebdfbcddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:58.871203 4947 tablet_bootstrap.cc:492] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: Bootstrap starting.
I20260928 21:58:58.871207 4943 tablet_bootstrap.cc:492] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: Bootstrap starting.
I20260928 21:58:58.871294 4852 tablet_service.cc:1467] Processing CreateTablet for tablet 5967eceb62a3451c82621daebdfbcddc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16221ac2b88945c29469c7f9370427a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:58:58.871436 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5967eceb62a3451c82621daebdfbcddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:58.872063 4943 tablet_bootstrap.cc:654] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:58.872063 4947 tablet_bootstrap.cc:654] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:58.872453 4941 tablet_bootstrap.cc:492] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: Bootstrap starting.
I20260928 21:58:58.873027 4941 tablet_bootstrap.cc:654] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:58.873754 4947 tablet_bootstrap.cc:492] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: No bootstrap required, opened a new log
I20260928 21:58:58.873759 4941 tablet_bootstrap.cc:492] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: No bootstrap required, opened a new log
I20260928 21:58:58.873822 4947 ts_tablet_manager.cc:1397] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:58:58.873849 4941 ts_tablet_manager.cc:1397] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:58.873888 4943 tablet_bootstrap.cc:492] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: No bootstrap required, opened a new log
I20260928 21:58:58.873948 4943 ts_tablet_manager.cc:1397] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:58:58.873998 4947 raft_consensus.cc:348] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:58.874011 4941 raft_consensus.cc:348] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:58.874090 4947 raft_consensus.cc:374] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:58.874115 4941 raft_consensus.cc:374] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:58.874119 4947 raft_consensus.cc:729] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5750dc33100a461e8703a63da45a60f3, State: Initialized, Role: FOLLOWER
I20260928 21:58:58.874146 4941 raft_consensus.cc:729] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931383f03b0c4c62a3a3599ecb1d922c, State: Initialized, Role: FOLLOWER
I20260928 21:58:58.874096 4943 raft_consensus.cc:348] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:58.874168 4943 raft_consensus.cc:374] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:58.874176 4947 consensus_queue.cc:260] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:58.874189 4941 consensus_queue.cc:260] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:58.874215 4943 raft_consensus.cc:729] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Initialized, Role: FOLLOWER
I20260928 21:58:58.874279 4941 ts_tablet_manager.cc:1428] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:58.874279 4947 ts_tablet_manager.cc:1428] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:58.874264 4943 consensus_queue.cc:260] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:58.874362 4943 ts_tablet_manager.cc:1428] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:58.874456 4949 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:58.874548 4949 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:58.874519 4946 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:58.874603 4946 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:58.874739 4944 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:58.874850 4944 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 21:58:58.890950 4639 tablet.cc:2369] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:58:58.972075 4919 tablet.cc:2369] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:59.033412 4946 raft_consensus.cc:482] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:59.033521 4946 raft_consensus.cc:504] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.033718 4946 leader_election.cc:288] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:58:59.033967 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967eceb62a3451c82621daebdfbcddc" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3" is_pre_election: true
I20260928 21:58:59.033969 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967eceb62a3451c82621daebdfbcddc" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c" is_pre_election: true
I20260928 21:58:59.034075 4872 raft_consensus.cc:2454] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:58:59.034075 4732 raft_consensus.cc:2454] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:58:59.034242 4523 leader_election.cc:302] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5750dc33100a461e8703a63da45a60f3, 8c6dc716658f45b0914bf55849910fdb; no voters:
I20260928 21:58:59.034359 4946 raft_consensus.cc:2781] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:59.034400 4946 raft_consensus.cc:482] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:59.034425 4946 raft_consensus.cc:3037] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:59.035063 4946 raft_consensus.cc:504] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.035214 4946 leader_election.cc:288] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Requested vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:58:59.035364 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967eceb62a3451c82621daebdfbcddc" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3"
I20260928 21:58:59.035451 4732 raft_consensus.cc:3037] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:59.035419 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5967eceb62a3451c82621daebdfbcddc" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c"
I20260928 21:58:59.035488 4872 raft_consensus.cc:3037] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:59.036263 4732 raft_consensus.cc:2454] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:58:59.036263 4872 raft_consensus.cc:2454] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:58:59.036428 4523 leader_election.cc:302] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5750dc33100a461e8703a63da45a60f3, 8c6dc716658f45b0914bf55849910fdb; no voters:
I20260928 21:58:59.036533 4946 raft_consensus.cc:2781] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:59.036610 4946 raft_consensus.cc:686] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [term 1 LEADER]: Becoming Leader. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Running, Role: LEADER
I20260928 21:58:59.036661 4946 consensus_queue.cc:237] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.037006 4951 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: TxnStatusTablet state changed. Reason: New leader 8c6dc716658f45b0914bf55849910fdb. Latest consensus state: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:59.037086 4951 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:58:59.037283 4963 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:58:59.037459 4963 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:58:59.037700 4433 catalog_manager.cc:5495] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c6dc716658f45b0914bf55849910fdb (127.2.165.65). New cstate: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } health_report { overall_health: UNKNOWN } } }
I20260928 21:58:59.057344 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:59.054386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38345 (local address 127.0.0.1:35244)
0928 21:58:59.054452 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:59.054465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:59.054522 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:59.054771 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:59.054772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:59.054773 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:58:59.054830 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:59.054833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.055549 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.055550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:59.056297 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:59.056299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.056725 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.056726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:59.056800 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:59.056818 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:58:59.057134 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:58:59.057145 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:58:59.057196 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:58:59.057261 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 21:58:59.057348 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:59.054509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.2.165.65:38345)
0928 21:58:59.054577 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:59.054580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:59.054590 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:59.054631 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:59.054632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:59.054654 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:59.054763 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:59.054911 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.055461 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.056386 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.056618 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.056741 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:59.056939 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:59.057060 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:59.057163 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:59.057210 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 21:58:59.057233 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:58:59.059458 4872 raft_consensus.cc:1261] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:59.059494 4732 raft_consensus.cc:1261] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:59.059828 4951 consensus_queue.cc:1035] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:59.059995 4946 consensus_queue.cc:1035] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:59.061312 4944 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: TxnStatusTablet state changed. Reason: New leader 8c6dc716658f45b0914bf55849910fdb. Latest consensus state: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:59.061448 4944 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:59.061455 4949 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: TxnStatusTablet state changed. Reason: New leader 8c6dc716658f45b0914bf55849910fdb. Latest consensus state: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:59.061543 4949 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:59.062855 4951 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:59.062959 4951 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:58:59.062983 4966 mvcc.cc:205] Tried to move back new op lower bound from 7334431699184537600 to 7334431699094589440. Current Snapshot: MvccSnapshot[applied={T|T < 7334431699184537600}]
I20260928 21:58:59.063056 4951 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:59.063103 4951 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 8c6dc716658f45b0914bf55849910fdb: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:58:59.063771 4949 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:59.063905 4949 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 5750dc33100a461e8703a63da45a60f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:59.064329 4968 mvcc.cc:205] Tried to move back new op lower bound from 7334431699184537600 to 7334431699094589440. Current Snapshot: MvccSnapshot[applied={T|T < 7334431699184537600}]
I20260928 21:58:59.064476 4967 mvcc.cc:205] Tried to move back new op lower bound from 7334431699184537600 to 7334431699094589440. Current Snapshot: MvccSnapshot[applied={T|T < 7334431699184537600}]
I20260928 21:58:59.064445 4944 tablet_replica.cc:439] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } } }
I20260928 21:58:59.064640 4944 tablet_replica.cc:442] T 5967eceb62a3451c82621daebdfbcddc P 931383f03b0c4c62a3a3599ecb1d922c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:58:59.068322 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:59.065590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.2.165.65:38345)
0928 21:58:59.065672 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:59.065674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:59.065682 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:59.065716 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:59.065717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:59.065727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:59.065787 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:59.065922 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.066705 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.067540 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.067840 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.067913 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:59.068073 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:59.068108 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:59.068160 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:59.068220 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:58:59.068241 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:58:59.073737 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:59.071311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:43613 (local address 127.2.165.65:38345)
0928 21:58:59.071364 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:59.071366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:59.071426 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:58:59.071499 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:59.071500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:59.071509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:59.071614 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:59.071746 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.072185 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.073024 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.073206 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.073287 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:59.073470 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:59.073505 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:59.073564 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:59.073629 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:58:59.073645 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:58:59.073755 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:59.071231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38345 (local address 127.2.165.65:43613)
0928 21:58:59.071373 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:59.071383 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:59.071443 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:59.071619 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:59.071620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:59.071621 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:58:59.071691 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:59.071693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.072259 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.072260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:59.072932 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:59.072934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:59.073286 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:59.073287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:59.073353 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:59.073366 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:58:59.073565 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:58:59.073567 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:58:59.073619 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:58:59.073659 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260928 21:58:59.075917 4872 raft_consensus.cc:1261] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:59.075930 4732 raft_consensus.cc:1261] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:59.076112 4946 consensus_queue.cc:1035] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:59.076208 4951 consensus_queue.cc:1035] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:59.091169 4433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:59.091274 4433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:59.092005 4433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Sending DeleteTablet for 3 replicas of tablet a679645c9d1c47eb9491805424fd6416
I20260928 21:58:59.092435 4852 tablet_service.cc:1514] Processing DeleteTablet for tablet a679645c9d1c47eb9491805424fd6416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:59 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:44692
I20260928 21:58:59.092765 4982 tablet_replica.cc:330] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: stopping tablet replica
I20260928 21:58:59.092777 4571 tablet_service.cc:1514] Processing DeleteTablet for tablet a679645c9d1c47eb9491805424fd6416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:59 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35228
I20260928 21:58:59.092929 4982 raft_consensus.cc:2229] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:59.093036 4982 raft_consensus.cc:2258] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:59.093030 4712 tablet_service.cc:1514] Processing DeleteTablet for tablet a679645c9d1c47eb9491805424fd6416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:59 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:43186
I20260928 21:58:59.093241 4984 tablet_replica.cc:330] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: stopping tablet replica
I20260928 21:58:59.093309 4433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:59.093473 4433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:59.093389 4984 raft_consensus.cc:2229] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:59.093633 4982 ts_tablet_manager.cc:1905] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:59.093918 4984 raft_consensus.cc:2258] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:59.094432 4984 ts_tablet_manager.cc:1905] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:59.095042 4852 tablet_service.cc:1467] Processing CreateTablet for tablet 048da5f796ca4e359ddb6d31892fd2f1 (DEFAULT_TABLE table=test-table [id=12396cdf34394cbd842488d73293f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:59.095158 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048da5f796ca4e359ddb6d31892fd2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:59.095476 4571 tablet_service.cc:1467] Processing CreateTablet for tablet 048da5f796ca4e359ddb6d31892fd2f1 (DEFAULT_TABLE table=test-table [id=12396cdf34394cbd842488d73293f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:59.095629 4982 ts_tablet_manager.cc:1918] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:58:59.095677 4571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048da5f796ca4e359ddb6d31892fd2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:59.095705 4982 log.cc:1196] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/a679645c9d1c47eb9491805424fd6416
I20260928 21:58:59.095976 4982 ts_tablet_manager.cc:1939] T a679645c9d1c47eb9491805424fd6416 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting consensus metadata
I20260928 21:58:59.096803 4984 ts_tablet_manager.cc:1918] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:58:59.096949 4984 log.cc:1196] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/a679645c9d1c47eb9491805424fd6416
I20260928 21:58:59.097251 4984 ts_tablet_manager.cc:1939] T a679645c9d1c47eb9491805424fd6416 P 5750dc33100a461e8703a63da45a60f3: Deleting consensus metadata
I20260928 21:58:59.097343 4943 tablet_bootstrap.cc:492] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: Bootstrap starting.
I20260928 21:58:59.097792 4943 tablet_bootstrap.cc:654] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:59.098143 4941 tablet_bootstrap.cc:492] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: Bootstrap starting.
I20260928 21:58:59.098170 4422 catalog_manager.cc:4841] TS 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939): tablet a679645c9d1c47eb9491805424fd6416 (table test-table [id=7c661e4369d84ef2a2f89ae33602c2fc]) successfully deleted
I20260928 21:58:59.098376 4983 tablet_replica.cc:330] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: stopping tablet replica
I20260928 21:58:59.098449 4419 catalog_manager.cc:4841] TS 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977): tablet a679645c9d1c47eb9491805424fd6416 (table test-table [id=7c661e4369d84ef2a2f89ae33602c2fc]) successfully deleted
I20260928 21:58:59.098495 4983 raft_consensus.cc:2229] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:58:59.098531 4941 tablet_bootstrap.cc:654] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:59.098654 4983 raft_consensus.cc:2258] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:59.098645 4712 tablet_service.cc:1467] Processing CreateTablet for tablet 048da5f796ca4e359ddb6d31892fd2f1 (DEFAULT_TABLE table=test-table [id=12396cdf34394cbd842488d73293f78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:59.098750 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048da5f796ca4e359ddb6d31892fd2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:59.099116 4941 tablet_bootstrap.cc:492] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: No bootstrap required, opened a new log
I20260928 21:58:59.099169 4941 ts_tablet_manager.cc:1397] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:59.099334 4941 raft_consensus.cc:348] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.099409 4941 raft_consensus.cc:374] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:59.099437 4941 raft_consensus.cc:729] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931383f03b0c4c62a3a3599ecb1d922c, State: Initialized, Role: FOLLOWER
I20260928 21:58:59.099474 4941 consensus_queue.cc:260] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.099556 4941 ts_tablet_manager.cc:1428] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:59.099915 4983 ts_tablet_manager.cc:1905] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:59.100040 4947 tablet_bootstrap.cc:492] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: Bootstrap starting.
I20260928 21:58:59.101164 4947 tablet_bootstrap.cc:654] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:59.101799 4943 tablet_bootstrap.cc:492] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: No bootstrap required, opened a new log
I20260928 21:58:59.101854 4943 ts_tablet_manager.cc:1397] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 21:58:59.102013 4943 raft_consensus.cc:348] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.102089 4943 raft_consensus.cc:374] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:59.102109 4943 raft_consensus.cc:729] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Initialized, Role: FOLLOWER
I20260928 21:58:59.102150 4943 consensus_queue.cc:260] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.102250 4943 ts_tablet_manager.cc:1428] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:59.102483 4947 tablet_bootstrap.cc:492] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: No bootstrap required, opened a new log
I20260928 21:58:59.102535 4947 ts_tablet_manager.cc:1397] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:58:59.102726 4947 raft_consensus.cc:348] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.102803 4947 raft_consensus.cc:374] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:59.102833 4947 raft_consensus.cc:729] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5750dc33100a461e8703a63da45a60f3, State: Initialized, Role: FOLLOWER
I20260928 21:58:59.102871 4947 consensus_queue.cc:260] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.102967 4947 ts_tablet_manager.cc:1428] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:59.103252 4983 ts_tablet_manager.cc:1918] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:58:59.103324 4983 log.cc:1196] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/a679645c9d1c47eb9491805424fd6416
I20260928 21:58:59.103669 4983 ts_tablet_manager.cc:1939] T a679645c9d1c47eb9491805424fd6416 P 8c6dc716658f45b0914bf55849910fdb: Deleting consensus metadata
I20260928 21:58:59.104130 4420 catalog_manager.cc:4841] TS 8c6dc716658f45b0914bf55849910fdb (127.2.165.65:38345): tablet a679645c9d1c47eb9491805424fd6416 (table test-table [id=7c661e4369d84ef2a2f89ae33602c2fc]) successfully deleted
I20260928 21:58:59.111420 4951 raft_consensus.cc:482] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:59.111534 4951 raft_consensus.cc:504] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.111768 4951 leader_election.cc:288] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:58:59.111946 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048da5f796ca4e359ddb6d31892fd2f1" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3" is_pre_election: true
I20260928 21:58:59.111958 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048da5f796ca4e359ddb6d31892fd2f1" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c" is_pre_election: true
I20260928 21:58:59.112061 4732 raft_consensus.cc:2454] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:58:59.112077 4872 raft_consensus.cc:2454] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:58:59.112232 4526 leader_election.cc:302] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c6dc716658f45b0914bf55849910fdb, 931383f03b0c4c62a3a3599ecb1d922c; no voters:
I20260928 21:58:59.112334 4951 raft_consensus.cc:2781] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:59.112370 4951 raft_consensus.cc:482] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:59.112391 4951 raft_consensus.cc:3037] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:59.113157 4951 raft_consensus.cc:504] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.113312 4951 leader_election.cc:288] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Requested vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:58:59.113529 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048da5f796ca4e359ddb6d31892fd2f1" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c"
I20260928 21:58:59.113544 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048da5f796ca4e359ddb6d31892fd2f1" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3"
I20260928 21:58:59.113613 4872 raft_consensus.cc:3037] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:59.113644 4732 raft_consensus.cc:3037] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:59.114194 4872 raft_consensus.cc:2454] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:58:59.114372 4526 leader_election.cc:302] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c6dc716658f45b0914bf55849910fdb, 931383f03b0c4c62a3a3599ecb1d922c; no voters:
I20260928 21:58:59.114473 4951 raft_consensus.cc:2781] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:59.114468 4732 raft_consensus.cc:2454] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:58:59.114518 4951 raft_consensus.cc:686] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 1 LEADER]: Becoming Leader. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Running, Role: LEADER
I20260928 21:58:59.114583 4951 consensus_queue.cc:237] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:58:59.115278 4434 catalog_manager.cc:5495] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c6dc716658f45b0914bf55849910fdb (127.2.165.65). New cstate: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } health_report { overall_health: UNKNOWN } } }
I20260928 21:58:59.122907 4732 raft_consensus.cc:1261] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:59.122920 4872 raft_consensus.cc:1261] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:59.123124 4989 consensus_queue.cc:1035] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:59.123250 4946 consensus_queue.cc:1035] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:58:59.128176 4966 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:59:00.646095 4434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:00.646202 4434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:00.647047 4434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Sending DeleteTablet for 3 replicas of tablet 048da5f796ca4e359ddb6d31892fd2f1
I20260928 21:59:00.647294 4572 tablet_service.cc:1514] Processing DeleteTablet for tablet 048da5f796ca4e359ddb6d31892fd2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:00 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35228
I20260928 21:59:00.647320 4852 tablet_service.cc:1514] Processing DeleteTablet for tablet 048da5f796ca4e359ddb6d31892fd2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:00 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:44692
I20260928 21:59:00.647481 5012 tablet_replica.cc:330] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: stopping tablet replica
I20260928 21:59:00.647536 5012 raft_consensus.cc:2229] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:00.647524 4712 tablet_service.cc:1514] Processing DeleteTablet for tablet 048da5f796ca4e359ddb6d31892fd2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:00 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:43186
I20260928 21:59:00.647594 5012 raft_consensus.cc:2258] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:00.647749 5014 tablet_replica.cc:330] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: stopping tablet replica
I20260928 21:59:00.647831 5014 raft_consensus.cc:2229] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:00.647884 5014 raft_consensus.cc:2258] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:00.647943 5012 ts_tablet_manager.cc:1905] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:00.648103 5014 ts_tablet_manager.cc:1905] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:00.649706 5013 tablet_replica.cc:330] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: stopping tablet replica
I20260928 21:59:00.649803 5013 raft_consensus.cc:2229] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:00.649848 5013 raft_consensus.cc:2258] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:00.650022 5012 ts_tablet_manager.cc:1918] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:59:00.650127 5013 ts_tablet_manager.cc:1905] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:00.650269 5014 ts_tablet_manager.cc:1918] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:59:00.650384 5014 log.cc:1196] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/048da5f796ca4e359ddb6d31892fd2f1
I20260928 21:59:00.650651 5014 ts_tablet_manager.cc:1939] T 048da5f796ca4e359ddb6d31892fd2f1 P 5750dc33100a461e8703a63da45a60f3: Deleting consensus metadata
I20260928 21:59:00.651027 4419 catalog_manager.cc:4841] TS 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977): tablet 048da5f796ca4e359ddb6d31892fd2f1 (table test-table [id=12396cdf34394cbd842488d73293f78b]) successfully deleted
I20260928 21:59:00.651217 5012 log.cc:1196] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/048da5f796ca4e359ddb6d31892fd2f1
I20260928 21:59:00.651675 5012 ts_tablet_manager.cc:1939] T 048da5f796ca4e359ddb6d31892fd2f1 P 8c6dc716658f45b0914bf55849910fdb: Deleting consensus metadata
I20260928 21:59:00.651867 5013 ts_tablet_manager.cc:1918] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:59:00.651928 5013 log.cc:1196] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/048da5f796ca4e359ddb6d31892fd2f1
I20260928 21:59:00.652117 4420 catalog_manager.cc:4841] TS 8c6dc716658f45b0914bf55849910fdb (127.2.165.65:38345): tablet 048da5f796ca4e359ddb6d31892fd2f1 (table test-table [id=12396cdf34394cbd842488d73293f78b]) successfully deleted
I20260928 21:59:00.652141 5013 ts_tablet_manager.cc:1939] T 048da5f796ca4e359ddb6d31892fd2f1 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting consensus metadata
I20260928 21:59:00.652763 4422 catalog_manager.cc:4841] TS 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939): tablet 048da5f796ca4e359ddb6d31892fd2f1 (table test-table [id=12396cdf34394cbd842488d73293f78b]) successfully deleted
Sep 28 21:59:00 dist-test-slave-3t36 krb5kdc[4390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:59:00 dist-test-slave-3t36 krb5kdc[4390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:00.683367 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.663959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.2.165.126:40139)
0928 21:59:00.664123 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:00.664125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:00.664139 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:59:00.664164 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:00.664164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:00.664174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:00.664268 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:00.664906 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.665529 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.666376 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.666590 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.666680 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.679860 (+ 13180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:00.679863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:00.679864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:00.679877 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:00.681902 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:00.682373 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:00.682374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:00.682374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:00.682428 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:00.682652 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:00.682654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:00.682654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:00.682828 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:00.682946 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:00.683065 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:59:00.683156 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260928 21:59:00.685081 4434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:00.685242 4434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:00.686659 4572 tablet_service.cc:1467] Processing CreateTablet for tablet b429cecfc5d14b62a4529058b831d751 (DEFAULT_TABLE table=test-table [id=66f513e9c4ed454cac7631b41f8a28f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:00.686684 4852 tablet_service.cc:1467] Processing CreateTablet for tablet b429cecfc5d14b62a4529058b831d751 (DEFAULT_TABLE table=test-table [id=66f513e9c4ed454cac7631b41f8a28f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:00.686703 4712 tablet_service.cc:1467] Processing CreateTablet for tablet b429cecfc5d14b62a4529058b831d751 (DEFAULT_TABLE table=test-table [id=66f513e9c4ed454cac7631b41f8a28f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:00.686821 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b429cecfc5d14b62a4529058b831d751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:00.686821 4572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b429cecfc5d14b62a4529058b831d751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:00.686866 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b429cecfc5d14b62a4529058b831d751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:00.688227 5025 tablet_bootstrap.cc:492] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: Bootstrap starting.
I20260928 21:59:00.688314 5023 tablet_bootstrap.cc:492] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: Bootstrap starting.
I20260928 21:59:00.688629 5025 tablet_bootstrap.cc:654] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:00.688647 5023 tablet_bootstrap.cc:654] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:00.688993 5024 tablet_bootstrap.cc:492] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: Bootstrap starting.
I20260928 21:59:00.689235 5023 tablet_bootstrap.cc:492] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: No bootstrap required, opened a new log
I20260928 21:59:00.689277 5023 ts_tablet_manager.cc:1397] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:00.689385 5023 raft_consensus.cc:348] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.689442 5023 raft_consensus.cc:374] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:00.689455 5023 raft_consensus.cc:729] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5750dc33100a461e8703a63da45a60f3, State: Initialized, Role: FOLLOWER
I20260928 21:59:00.689435 5024 tablet_bootstrap.cc:654] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:00.689517 5023 consensus_queue.cc:260] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.689668 5023 ts_tablet_manager.cc:1428] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:00.690006 5025 tablet_bootstrap.cc:492] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: No bootstrap required, opened a new log
I20260928 21:59:00.690064 5025 ts_tablet_manager.cc:1397] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:00.690176 5025 raft_consensus.cc:348] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.690264 5025 raft_consensus.cc:374] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:00.690292 5025 raft_consensus.cc:729] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Initialized, Role: FOLLOWER
I20260928 21:59:00.690351 5025 consensus_queue.cc:260] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.690505 5025 ts_tablet_manager.cc:1428] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:00.690753 5024 tablet_bootstrap.cc:492] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: No bootstrap required, opened a new log
I20260928 21:59:00.690809 5024 ts_tablet_manager.cc:1397] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:00.690945 5024 raft_consensus.cc:348] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.691043 5024 raft_consensus.cc:374] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:00.691080 5024 raft_consensus.cc:729] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931383f03b0c4c62a3a3599ecb1d922c, State: Initialized, Role: FOLLOWER
I20260928 21:59:00.691138 5024 consensus_queue.cc:260] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.691320 5024 ts_tablet_manager.cc:1428] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:00.786543 5000 raft_consensus.cc:482] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:00.786693 5000 raft_consensus.cc:504] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.786911 5000 leader_election.cc:288] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:59:00.787173 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b429cecfc5d14b62a4529058b831d751" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c" is_pre_election: true
I20260928 21:59:00.787173 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b429cecfc5d14b62a4529058b831d751" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3" is_pre_election: true
I20260928 21:59:00.787294 4732 raft_consensus.cc:2454] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:59:00.787292 4872 raft_consensus.cc:2454] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 0.
I20260928 21:59:00.787477 4526 leader_election.cc:302] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c6dc716658f45b0914bf55849910fdb, 931383f03b0c4c62a3a3599ecb1d922c; no voters:
I20260928 21:59:00.787577 5000 raft_consensus.cc:2781] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:00.787619 5000 raft_consensus.cc:482] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:00.787647 5000 raft_consensus.cc:3037] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:00.788378 5000 raft_consensus.cc:504] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.788524 5000 leader_election.cc:288] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Requested vote from peers 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:59:00.788695 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b429cecfc5d14b62a4529058b831d751" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5750dc33100a461e8703a63da45a60f3"
I20260928 21:59:00.788733 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b429cecfc5d14b62a4529058b831d751" candidate_uuid: "8c6dc716658f45b0914bf55849910fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c"
I20260928 21:59:00.788791 4732 raft_consensus.cc:3037] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:00.788848 4872 raft_consensus.cc:3037] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:00.789445 4732 raft_consensus.cc:2454] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:59:00.789476 4872 raft_consensus.cc:2454] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c6dc716658f45b0914bf55849910fdb in term 1.
I20260928 21:59:00.789629 4523 leader_election.cc:302] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5750dc33100a461e8703a63da45a60f3, 8c6dc716658f45b0914bf55849910fdb; no voters:
I20260928 21:59:00.789764 5000 raft_consensus.cc:2781] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:00.789834 5000 raft_consensus.cc:686] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 1 LEADER]: Becoming Leader. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Running, Role: LEADER
I20260928 21:59:00.789897 5000 consensus_queue.cc:237] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.790547 4433 catalog_manager.cc:5495] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c6dc716658f45b0914bf55849910fdb (127.2.165.65). New cstate: current_term: 1 leader_uuid: "8c6dc716658f45b0914bf55849910fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:00.810276 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.807434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.2.165.65:38345)
0928 21:59:00.807581 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:00.807583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:00.807590 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:59:00.807610 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:00.807611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:00.807621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:00.807677 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:00.807905 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.808383 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.809300 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.809505 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.809563 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.809771 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:00.809802 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:00.809855 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:00.810009 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 21:59:00.810161 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260928 21:59:00.812618 4872 raft_consensus.cc:1261] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:00.812602 4732 raft_consensus.cc:1261] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Refusing update from remote peer 8c6dc716658f45b0914bf55849910fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:00.812891 4995 consensus_queue.cc:1035] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:00.812989 5000 consensus_queue.cc:1035] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:00.823215 4434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:00.823333 4434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:00.824033 4434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Sending DeleteTablet for 3 replicas of tablet b429cecfc5d14b62a4529058b831d751
I20260928 21:59:00.824350 4852 tablet_service.cc:1514] Processing DeleteTablet for tablet b429cecfc5d14b62a4529058b831d751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:00 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:44692
I20260928 21:59:00.824380 4712 tablet_service.cc:1514] Processing DeleteTablet for tablet b429cecfc5d14b62a4529058b831d751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:00 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:43186
I20260928 21:59:00.824465 5013 tablet_replica.cc:330] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: stopping tablet replica
I20260928 21:59:00.824492 5014 tablet_replica.cc:330] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: stopping tablet replica
I20260928 21:59:00.824469 4572 tablet_service.cc:1514] Processing DeleteTablet for tablet b429cecfc5d14b62a4529058b831d751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:00 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35228
I20260928 21:59:00.824520 5013 raft_consensus.cc:2229] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:00.824536 5014 raft_consensus.cc:2229] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:00.824551 5012 tablet_replica.cc:330] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: stopping tablet replica
I20260928 21:59:00.824589 5013 raft_consensus.cc:2258] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:00.824592 5014 raft_consensus.cc:2258] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:00.824594 5012 raft_consensus.cc:2229] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:00.824652 5012 raft_consensus.cc:2258] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:00.824952 5013 ts_tablet_manager.cc:1905] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:00.824990 5014 ts_tablet_manager.cc:1905] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:00.825031 4434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:59:00.825146 5012 ts_tablet_manager.cc:1905] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:59:00.825181 4434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:00.826673 5012 ts_tablet_manager.cc:1918] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:00.826787 5012 log.cc:1196] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/b429cecfc5d14b62a4529058b831d751
I20260928 21:59:00.826923 5014 ts_tablet_manager.cc:1918] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:00.827020 5014 log.cc:1196] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/b429cecfc5d14b62a4529058b831d751
I20260928 21:59:00.827047 5012 ts_tablet_manager.cc:1939] T b429cecfc5d14b62a4529058b831d751 P 8c6dc716658f45b0914bf55849910fdb: Deleting consensus metadata
I20260928 21:59:00.827245 5014 ts_tablet_manager.cc:1939] T b429cecfc5d14b62a4529058b831d751 P 5750dc33100a461e8703a63da45a60f3: Deleting consensus metadata
I20260928 21:59:00.827286 5013 ts_tablet_manager.cc:1918] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:00.827370 5013 log.cc:1196] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/b429cecfc5d14b62a4529058b831d751
I20260928 21:59:00.827456 4852 tablet_service.cc:1467] Processing CreateTablet for tablet 23d97af47bd74ae482a56e550c89b31e (DEFAULT_TABLE table=test-table [id=e4aa2276ec064c3aa6e141fa5e83d008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:00.827559 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d97af47bd74ae482a56e550c89b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:00.827659 5013 ts_tablet_manager.cc:1939] T b429cecfc5d14b62a4529058b831d751 P 931383f03b0c4c62a3a3599ecb1d922c: Deleting consensus metadata
I20260928 21:59:00.828020 4422 catalog_manager.cc:4841] TS 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939): tablet b429cecfc5d14b62a4529058b831d751 (table test-table [id=66f513e9c4ed454cac7631b41f8a28f0]) successfully deleted
I20260928 21:59:00.828543 5024 tablet_bootstrap.cc:492] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: Bootstrap starting.
I20260928 21:59:00.828701 4420 catalog_manager.cc:4841] TS 8c6dc716658f45b0914bf55849910fdb (127.2.165.65:38345): tablet b429cecfc5d14b62a4529058b831d751 (table test-table [id=66f513e9c4ed454cac7631b41f8a28f0]) successfully deleted
I20260928 21:59:00.828797 4572 tablet_service.cc:1467] Processing CreateTablet for tablet 23d97af47bd74ae482a56e550c89b31e (DEFAULT_TABLE table=test-table [id=e4aa2276ec064c3aa6e141fa5e83d008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:00.828899 4572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d97af47bd74ae482a56e550c89b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:00.828914 4419 catalog_manager.cc:4841] TS 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977): tablet b429cecfc5d14b62a4529058b831d751 (table test-table [id=66f513e9c4ed454cac7631b41f8a28f0]) successfully deleted
I20260928 21:59:00.828948 5024 tablet_bootstrap.cc:654] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:00.829013 4712 tablet_service.cc:1467] Processing CreateTablet for tablet 23d97af47bd74ae482a56e550c89b31e (DEFAULT_TABLE table=test-table [id=e4aa2276ec064c3aa6e141fa5e83d008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:00.829097 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d97af47bd74ae482a56e550c89b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:00.829538 5024 tablet_bootstrap.cc:492] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: No bootstrap required, opened a new log
I20260928 21:59:00.829593 5024 ts_tablet_manager.cc:1397] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:00.829749 5024 raft_consensus.cc:348] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.829816 5024 raft_consensus.cc:374] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:00.829835 5024 raft_consensus.cc:729] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931383f03b0c4c62a3a3599ecb1d922c, State: Initialized, Role: FOLLOWER
I20260928 21:59:00.829911 5024 consensus_queue.cc:260] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.830034 5024 ts_tablet_manager.cc:1428] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:00.830030 5023 tablet_bootstrap.cc:492] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: Bootstrap starting.
I20260928 21:59:00.830202 5025 tablet_bootstrap.cc:492] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb: Bootstrap starting.
I20260928 21:59:00.830524 5025 tablet_bootstrap.cc:654] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:00.830641 5023 tablet_bootstrap.cc:654] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:00.831063 5025 tablet_bootstrap.cc:492] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb: No bootstrap required, opened a new log
I20260928 21:59:00.831120 5025 ts_tablet_manager.cc:1397] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:00.831251 5025 raft_consensus.cc:348] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.831324 5025 raft_consensus.cc:374] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:00.831351 5025 raft_consensus.cc:729] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6dc716658f45b0914bf55849910fdb, State: Initialized, Role: FOLLOWER
I20260928 21:59:00.831404 5025 consensus_queue.cc:260] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.831497 5025 ts_tablet_manager.cc:1428] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:00.832036 5023 tablet_bootstrap.cc:492] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: No bootstrap required, opened a new log
I20260928 21:59:00.832114 5023 ts_tablet_manager.cc:1397] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:59:00.832255 5023 raft_consensus.cc:348] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.832332 5023 raft_consensus.cc:374] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:00.832347 5023 raft_consensus.cc:729] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5750dc33100a461e8703a63da45a60f3, State: Initialized, Role: FOLLOWER
I20260928 21:59:00.832417 5023 consensus_queue.cc:260] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.832509 5023 ts_tablet_manager.cc:1428] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:00.838909 5028 raft_consensus.cc:482] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:00.838997 5028 raft_consensus.cc:504] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.839325 5028 leader_election.cc:288] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c6dc716658f45b0914bf55849910fdb (127.2.165.65:38345), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:59:00.843588 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.839528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:35299 (local address 127.2.165.67:37939)
0928 21:59:00.839724 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:00.839727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:00.839734 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:59:00.839757 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:00.839758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:00.839768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:00.839843 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:00.840070 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.840845 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.842540 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.843152 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.843223 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.843231 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:00.843458 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 21:59:00.843510 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
I20260928 21:59:00.843580 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.839641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:45373 (local address 127.2.165.65:38345)
0928 21:59:00.839699 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:00.839702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:00.839712 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:00.839757 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:00.839757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:00.839768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:00.839836 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:00.840070 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.840846 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.842540 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.843119 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.843205 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.843217 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:00.843460 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 21:59:00.843505 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:59:00.843622 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.839441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38345 (local address 127.2.165.66:45373)
0928 21:59:00.839570 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:00.839584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:00.839644 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:00.839863 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:00.839864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:00.839868 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:00.839989 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.839992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.840993 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.840994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.842445 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.842447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.843239 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.843240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.843347 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.843358 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:59:00.843423 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:59:00.843478 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":102,"thread_start_us":84,"threads_started":1}
I20260928 21:59:00.843626 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.839459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:37939 (local address 127.2.165.66:35299)
0928 21:59:00.839591 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:00.839600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:00.839644 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:00.839863 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:00.839864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:00.839867 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:00.839989 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.839991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.840968 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.840970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.842398 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.842402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.843241 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.843242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.843347 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.843359 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 21:59:00.843423 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:59:00.843478 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":46,"thread_start_us":84,"threads_started":1}
I20260928 21:59:00.843979 4592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d97af47bd74ae482a56e550c89b31e" candidate_uuid: "5750dc33100a461e8703a63da45a60f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c6dc716658f45b0914bf55849910fdb" is_pre_election: true
I20260928 21:59:00.844138 4592 raft_consensus.cc:2454] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5750dc33100a461e8703a63da45a60f3 in term 0.
I20260928 21:59:00.844233 4871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d97af47bd74ae482a56e550c89b31e" candidate_uuid: "5750dc33100a461e8703a63da45a60f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c" is_pre_election: true
I20260928 21:59:00.844342 4871 raft_consensus.cc:2454] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5750dc33100a461e8703a63da45a60f3 in term 0.
I20260928 21:59:00.844340 4664 leader_election.cc:302] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5750dc33100a461e8703a63da45a60f3, 8c6dc716658f45b0914bf55849910fdb; no voters:
I20260928 21:59:00.844499 5028 raft_consensus.cc:2781] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:00.844549 5028 raft_consensus.cc:482] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:00.844574 5028 raft_consensus.cc:3037] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:00.845357 5028 raft_consensus.cc:504] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.845528 5028 leader_election.cc:288] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [CANDIDATE]: Term 1 election: Requested vote from peers 8c6dc716658f45b0914bf55849910fdb (127.2.165.65:38345), 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939)
I20260928 21:59:00.845685 4592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d97af47bd74ae482a56e550c89b31e" candidate_uuid: "5750dc33100a461e8703a63da45a60f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c6dc716658f45b0914bf55849910fdb"
I20260928 21:59:00.845675 4871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d97af47bd74ae482a56e550c89b31e" candidate_uuid: "5750dc33100a461e8703a63da45a60f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931383f03b0c4c62a3a3599ecb1d922c"
I20260928 21:59:00.845752 4592 raft_consensus.cc:3037] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:00.845752 4871 raft_consensus.cc:3037] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:00.846519 4871 raft_consensus.cc:2454] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5750dc33100a461e8703a63da45a60f3 in term 1.
I20260928 21:59:00.846519 4592 raft_consensus.cc:2454] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5750dc33100a461e8703a63da45a60f3 in term 1.
I20260928 21:59:00.846680 4664 leader_election.cc:302] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5750dc33100a461e8703a63da45a60f3, 8c6dc716658f45b0914bf55849910fdb; no voters:
I20260928 21:59:00.846781 5028 raft_consensus.cc:2781] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:00.846930 5028 raft_consensus.cc:686] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 1 LEADER]: Becoming Leader. State: Replica: 5750dc33100a461e8703a63da45a60f3, State: Running, Role: LEADER
I20260928 21:59:00.847009 5028 consensus_queue.cc:237] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } }
I20260928 21:59:00.847751 4434 catalog_manager.cc:5495] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5750dc33100a461e8703a63da45a60f3 (127.2.165.66). New cstate: current_term: 1 leader_uuid: "5750dc33100a461e8703a63da45a60f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5750dc33100a461e8703a63da45a60f3" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:00.858126 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.855239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.2.165.66:42977)
0928 21:59:00.855452 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:00.855454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:00.855460 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:59:00.855495 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:00.855496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:00.855515 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:00.855611 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:00.855758 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.856375 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.857248 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.857463 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.857539 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.857734 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:00.857857 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:00.857945 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:00.857984 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 21:59:00.858007 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":135,"threads_started":1}
I20260928 21:59:00.863565 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.861160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:56151 (local address 127.2.165.65:38345)
0928 21:59:00.861216 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:00.861218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:00.861308 (+ 90us) server_negotiation.cc:407] Connection header received
0928 21:59:00.861396 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:00.861397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:00.861408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:00.861468 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:00.861592 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.862109 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.862877 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.863124 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.863175 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.863333 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:00.863395 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:00.863452 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:00.863487 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:59:00.863505 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:59:00.863577 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.861066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38345 (local address 127.2.165.66:56151)
0928 21:59:00.861231 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:00.861248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:00.861320 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:00.861485 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:00.861485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:00.861487 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:59:00.861543 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.861544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.862172 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.862172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.862803 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.862805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.863176 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.863177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.863244 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.863257 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:59:00.863454 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:59:00.863455 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:59:00.863484 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 21:59:00.863514 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260928 21:59:00.866058 4592 raft_consensus.cc:1261] T 23d97af47bd74ae482a56e550c89b31e P 8c6dc716658f45b0914bf55849910fdb [term 1 FOLLOWER]: Refusing update from remote peer 5750dc33100a461e8703a63da45a60f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:00.866050 4871 raft_consensus.cc:1261] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Refusing update from remote peer 5750dc33100a461e8703a63da45a60f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:00.866346 5028 consensus_queue.cc:1035] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c6dc716658f45b0914bf55849910fdb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:00.866505 5044 consensus_queue.cc:1035] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "931383f03b0c4c62a3a3599ecb1d922c" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:00.873718 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.871196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:51723 (local address 127.2.165.66:42977)
0928 21:59:00.871273 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:00.871275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:00.871374 (+ 99us) server_negotiation.cc:407] Connection header received
0928 21:59:00.871439 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:00.871440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:00.871452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:00.871501 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:00.871610 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.872103 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.873002 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.873218 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.873298 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.873455 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:00.873492 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:00.873541 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:00.873615 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:59:00.873637 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:59:00.873734 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:00.871164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42977 (local address 127.2.165.65:51723)
0928 21:59:00.871288 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:00.871299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:00.871388 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:00.871503 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:00.871504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:00.871506 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:59:00.871560 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.871562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.872236 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.872237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.872920 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:00.872923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:00.873296 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:00.873296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:00.873364 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:00.873369 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:59:00.873556 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:59:00.873558 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:59:00.873609 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:59:00.873659 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
W20260928 21:59:00.874212 4997 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:59:02.400825 4434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:02.400962 4434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:02.402051 4434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f4547de5b674602a2e1550f53655554: Sending DeleteTablet for 3 replicas of tablet 23d97af47bd74ae482a56e550c89b31e
I20260928 21:59:02.402427 4712 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d97af47bd74ae482a56e550c89b31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:02 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:43186
I20260928 21:59:02.402478 4852 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d97af47bd74ae482a56e550c89b31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:02 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:44692
I20260928 21:59:02.402635 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 4507
I20260928 21:59:02.402660 4572 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d97af47bd74ae482a56e550c89b31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:02 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35228
I20260928 21:59:02.403431 5079 tablet_replica.cc:330] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: stopping tablet replica
I20260928 21:59:02.403501 5079 raft_consensus.cc:2229] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:02.403545 5079 raft_consensus.cc:2258] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:02.403587 5078 tablet_replica.cc:330] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: stopping tablet replica
I20260928 21:59:02.403651 5078 raft_consensus.cc:2229] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:02.403750 5078 raft_consensus.cc:2258] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:02.403822 5079 ts_tablet_manager.cc:1905] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:02.404031 5078 ts_tablet_manager.cc:1905] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:02.406203 5078 ts_tablet_manager.cc:1918] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:59:02.406237 5079 ts_tablet_manager.cc:1918] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:59:02.406280 5078 log.cc:1196] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/23d97af47bd74ae482a56e550c89b31e
I20260928 21:59:02.406323 5079 log.cc:1196] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/23d97af47bd74ae482a56e550c89b31e
I20260928 21:59:02.406558 5079 ts_tablet_manager.cc:1939] T 23d97af47bd74ae482a56e550c89b31e P 931383f03b0c4c62a3a3599ecb1d922c: Deleting consensus metadata
I20260928 21:59:02.406560 5078 ts_tablet_manager.cc:1939] T 23d97af47bd74ae482a56e550c89b31e P 5750dc33100a461e8703a63da45a60f3: Deleting consensus metadata
I20260928 21:59:02.406953 4419 catalog_manager.cc:4841] TS 5750dc33100a461e8703a63da45a60f3 (127.2.165.66:42977): tablet 23d97af47bd74ae482a56e550c89b31e (table test-table [id=e4aa2276ec064c3aa6e141fa5e83d008]) successfully deleted
I20260928 21:59:02.408856 4422 catalog_manager.cc:4841] TS 931383f03b0c4c62a3a3599ecb1d922c (127.2.165.67:37939): tablet 23d97af47bd74ae482a56e550c89b31e (table test-table [id=e4aa2276ec064c3aa6e141fa5e83d008]) successfully deleted
W20260928 21:59:02.410486 4420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:38345 (error 108)
I20260928 21:59:02.410791 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 4647
I20260928 21:59:02.412520 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:02.410966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38345 (local address 127.0.0.1:52140)
0928 21:59:02.412433 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:02.412482 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1441,"thread_start_us":81,"threads_started":1}
W20260928 21:59:02.412650 4420 catalog_manager.cc:4584] TS 8c6dc716658f45b0914bf55849910fdb (127.2.165.65:38345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23d97af47bd74ae482a56e550c89b31e: Network error: Client connection negotiation failed: client connection to 127.2.165.65:38345: connect: Connection refused (error 111)
I20260928 21:59:02.416978 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 4787
I20260928 21:59:02.421764 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 4403
2026-09-28T21:59:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5587 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:02 dist-test-slave-3t36 krb5kdc[5084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:02 dist-test-slave-3t36 krb5kdc[5084](info): set up 2 sockets
Sep 28 21:59:02 dist-test-slave-3t36 krb5kdc[5084](info): commencing operation
krb5kdc: starting...
W20260928 21:59:03.460530 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:03 dist-test-slave-3t36 krb5kdc[5084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:03.585119 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:44749
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:03.668026 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:03.668260 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:03.668284 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:03.669440 5097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:03.669480 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:03.669494 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:03.669507 5097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:03.669518 5097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:03.670956 5097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:03.671162 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:03.671341 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:03.676555 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:03.676592 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:03.676784 5097 server_base.cc:1034] running on GCE node
W20260928 21:59:03.676550 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:03.677191 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:03.677456 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:03.678597 5097 hybrid_clock.cc:648] HybridClock initialized: now 1790632743678577 us; error 36 us; skew 500 ppm
Sep 28 21:59:03 dist-test-slave-3t36 krb5kdc[5084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:03.681226 5097 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:03.681659 5097 webserver.cc:460] Webserver started at http://127.2.165.126:40997/ using document root <none> and password file <none>
I20260928 21:59:03.681897 5097 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:03.681974 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:03.682120 5097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:03.683207 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "e840088bca6045418761a8667664ac6b"
format_stamp: "Formatted at 2026-09-28 21:59:03 on dist-test-slave-3t36"
server_key: "5aac6bf3af4a49da0085759c748f50ef"
server_key_iv: "cd7fc1bdce1ff20eaead70ce746aed3e"
server_key_version: "encryptionkey@0"
I20260928 21:59:03.683584 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "e840088bca6045418761a8667664ac6b"
format_stamp: "Formatted at 2026-09-28 21:59:03 on dist-test-slave-3t36"
server_key: "5aac6bf3af4a49da0085759c748f50ef"
server_key_iv: "cd7fc1bdce1ff20eaead70ce746aed3e"
server_key_version: "encryptionkey@0"
I20260928 21:59:03.685034 5097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:03.685751 5112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:03.685933 5097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:03.686033 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "e840088bca6045418761a8667664ac6b"
format_stamp: "Formatted at 2026-09-28 21:59:03 on dist-test-slave-3t36"
server_key: "5aac6bf3af4a49da0085759c748f50ef"
server_key_iv: "cd7fc1bdce1ff20eaead70ce746aed3e"
server_key_version: "encryptionkey@0"
I20260928 21:59:03.686115 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:03.704509 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:03.706760 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:03.706964 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:03.711076 5097 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:44749
I20260928 21:59:03.711162 5164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:44749 every 8 connection(s)
I20260928 21:59:03.711498 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:03.712395 5165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:03.714706 5165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b: Bootstrap starting.
I20260928 21:59:03.715313 5165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:03.715610 5165 log.cc:824] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:03.716326 5165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b: No bootstrap required, opened a new log
I20260928 21:59:03.718029 5165 raft_consensus.cc:348] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840088bca6045418761a8667664ac6b" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44749 } }
I20260928 21:59:03.718165 5165 raft_consensus.cc:374] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:03.718187 5165 raft_consensus.cc:729] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e840088bca6045418761a8667664ac6b, State: Initialized, Role: FOLLOWER
I20260928 21:59:03.718364 5165 consensus_queue.cc:260] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840088bca6045418761a8667664ac6b" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44749 } }
I20260928 21:59:03.718425 5165 raft_consensus.cc:388] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:03.718446 5165 raft_consensus.cc:482] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:03.718513 5165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:03.719022 5165 raft_consensus.cc:504] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840088bca6045418761a8667664ac6b" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44749 } }
I20260928 21:59:03.719103 5165 leader_election.cc:302] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e840088bca6045418761a8667664ac6b; no voters:
I20260928 21:59:03.719310 5165 leader_election.cc:288] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:03.719354 5168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:03.719568 5165 sys_catalog.cc:564] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:03.719862 5168 raft_consensus.cc:686] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [term 1 LEADER]: Becoming Leader. State: Replica: e840088bca6045418761a8667664ac6b, State: Running, Role: LEADER
I20260928 21:59:03.719996 5168 consensus_queue.cc:237] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840088bca6045418761a8667664ac6b" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44749 } }
I20260928 21:59:03.720363 5170 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e840088bca6045418761a8667664ac6b. Latest consensus state: current_term: 1 leader_uuid: "e840088bca6045418761a8667664ac6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840088bca6045418761a8667664ac6b" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44749 } } }
I20260928 21:59:03.720430 5170 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:03.720496 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5097
I20260928 21:59:03.720589 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:03.720564 5169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e840088bca6045418761a8667664ac6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e840088bca6045418761a8667664ac6b" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44749 } } }
I20260928 21:59:03.720628 5169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:03.720700 5177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:03.720829 2709 external_mini_cluster.cc:1428] Setting key 708641d9856063f02aaf5fb65ea57ac5
I20260928 21:59:03.721216 5177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:03.723155 5177 catalog_manager.cc:1347] Generated new cluster ID: 9f44b25d32fb454db8099b19f49db8b3
I20260928 21:59:03.723222 5177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:59:03 dist-test-slave-3t36 krb5kdc[5084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:03.733700 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:03.721451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.2.165.126:44749)
0928 21:59:03.721748 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:03.721759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:03.721793 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:59:03.722091 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:03.722102 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:03.722174 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:03.722352 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:03.723106 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:03.724708 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:03.725822 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:03.726481 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:03.726586 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:03.729689 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:03.729695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:03.729695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:03.729709 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:03.731855 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:03.732287 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:03.732291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:03.732291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:03.732407 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:03.732627 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:03.732628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:03.732629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:03.732989 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:03.733172 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:03.733292 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 21:59:03.733447 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":75,"threads_started":1}
I20260928 21:59:03.739897 5177 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:03.740698 5177 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:03.756994 5177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e840088bca6045418761a8667664ac6b: Generated new TSK 0
I20260928 21:59:03.757247 5177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:59:03.792230 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44749
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:03.877377 5191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:03.877646 5191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:03.877693 5191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:03.879006 5191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:03.879086 5191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:03.879134 5191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:03.880781 5191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:03.881063 5191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:03.881313 5191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:03.886651 5199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:03.886672 5197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:03.886873 5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:03.886926 5191 server_base.cc:1034] running on GCE node
I20260928 21:59:03.887153 5191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:03.887377 5191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:03.888533 5191 hybrid_clock.cc:648] HybridClock initialized: now 1790632743888521 us; error 29 us; skew 500 ppm
Sep 28 21:59:03 dist-test-slave-3t36 krb5kdc[5084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:03.891453 5191 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:59:03.892040 5191 webserver.cc:460] Webserver started at http://127.2.165.65:44709/ using document root <none> and password file <none>
I20260928 21:59:03.892287 5191 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:03.892359 5191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:03.892524 5191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:03.893707 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "e0d7c5ce4ff54094a44b1fe76a4e02db"
format_stamp: "Formatted at 2026-09-28 21:59:03 on dist-test-slave-3t36"
server_key: "aaa9eebbcfecd2883508332239cfb8e2"
server_key_iv: "ab2d7225f3aeabc1ad35e9b693f4a7fc"
server_key_version: "encryptionkey@0"
I20260928 21:59:03.894100 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d7c5ce4ff54094a44b1fe76a4e02db"
format_stamp: "Formatted at 2026-09-28 21:59:03 on dist-test-slave-3t36"
server_key: "aaa9eebbcfecd2883508332239cfb8e2"
server_key_iv: "ab2d7225f3aeabc1ad35e9b693f4a7fc"
server_key_version: "encryptionkey@0"
I20260928 21:59:03.895514 5191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:03.896363 5206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:03.896548 5191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:03.896661 5191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "e0d7c5ce4ff54094a44b1fe76a4e02db"
format_stamp: "Formatted at 2026-09-28 21:59:03 on dist-test-slave-3t36"
server_key: "aaa9eebbcfecd2883508332239cfb8e2"
server_key_iv: "ab2d7225f3aeabc1ad35e9b693f4a7fc"
server_key_version: "encryptionkey@0"
I20260928 21:59:03.896790 5191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:03.919957 5191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:03.922689 5191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:03.922873 5191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:03.923141 5191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:03.923485 5191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:03.923545 5191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:03.923590 5191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:03.923620 5191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:03.929867 5191 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:33685
I20260928 21:59:03.929919 5319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:33685 every 8 connection(s)
I20260928 21:59:03.930261 5191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:59:03 dist-test-slave-3t36 krb5kdc[5084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:03.938520 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5191
I20260928 21:59:03.938638 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:03.938813 2709 external_mini_cluster.cc:1428] Setting key 8083c491e5c6f8a21f22190813e592c8
I20260928 21:59:03.940845 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:03.931069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:50223 (local address 127.2.165.126:44749)
0928 21:59:03.931155 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:03.931157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:03.931310 (+ 153us) server_negotiation.cc:407] Connection header received
0928 21:59:03.931850 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:03.931851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:03.931862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:03.931935 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:03.933071 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:03.933614 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:03.934681 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:03.934920 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:03.934986 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:03.937387 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:03.937389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:03.937389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:03.937401 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:03.939101 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:03.939555 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:03.939556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:03.939557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:03.939622 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:03.939955 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:03.939956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:03.939957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:03.940134 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:03.940200 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:03.940571 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 21:59:03.940651 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:59:03.941069 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:03.930990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:44749 (local address 127.2.165.65:50223)
0928 21:59:03.931174 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:03.931217 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:03.931727 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:03.932049 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:03.932053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:03.932724 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:03.932940 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:03.932945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:03.933753 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:03.933754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:03.934579 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:03.934581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:03.935032 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:03.935033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:03.935160 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:03.935697 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:03.935705 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:03.937188 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:03.939177 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:03.939185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:03.939188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:03.939481 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:03.939704 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:03.939705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:03.939706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:03.939867 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:03.940210 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:03.940216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:03.940421 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 21:59:03.940545 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 21:59:03.940773 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260928 21:59:03.941666 5320 heartbeater.cc:347] Connected to a master server at 127.2.165.126:44749
I20260928 21:59:03.941763 5320 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:03.941972 5320 heartbeater.cc:511] Master 127.2.165.126:44749 requested a full tablet report, sending...
I20260928 21:59:03.942550 5129 ts_manager.cc:194] Registered new tserver with Master: e0d7c5ce4ff54094a44b1fe76a4e02db (127.2.165.65:33685)
I20260928 21:59:03.943246 5129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:50223
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:59:03.965927 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44749
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:04.049028 5325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:04.049255 5325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:04.049283 5325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:04.050421 5325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:04.050460 5325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:04.050524 5325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:04.051981 5325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:04.052183 5325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:04.052368 5325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:04.057631 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:04.057834 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:04.057875 5330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:04.057957 5325 server_base.cc:1034] running on GCE node
I20260928 21:59:04.058185 5325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:04.058468 5325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:04.059656 5325 hybrid_clock.cc:648] HybridClock initialized: now 1790632744059625 us; error 41 us; skew 500 ppm
Sep 28 21:59:04 dist-test-slave-3t36 krb5kdc[5084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632744, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:04.062443 5325 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:59:04.062997 5325 webserver.cc:460] Webserver started at http://127.2.165.66:34341/ using document root <none> and password file <none>
I20260928 21:59:04.063235 5325 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:04.063309 5325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:04.063472 5325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:04.064666 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "7084698f3943403fb7974c4f001009ea"
format_stamp: "Formatted at 2026-09-28 21:59:04 on dist-test-slave-3t36"
server_key: "bae2377abfeabda2b98fa33bdd14c2f6"
server_key_iv: "611fe24036eb938e849746be0d641e26"
server_key_version: "encryptionkey@0"
I20260928 21:59:04.065160 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "7084698f3943403fb7974c4f001009ea"
format_stamp: "Formatted at 2026-09-28 21:59:04 on dist-test-slave-3t36"
server_key: "bae2377abfeabda2b98fa33bdd14c2f6"
server_key_iv: "611fe24036eb938e849746be0d641e26"
server_key_version: "encryptionkey@0"
I20260928 21:59:04.066628 5325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:04.067500 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:04.067708 5325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:04.067824 5325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "7084698f3943403fb7974c4f001009ea"
format_stamp: "Formatted at 2026-09-28 21:59:04 on dist-test-slave-3t36"
server_key: "bae2377abfeabda2b98fa33bdd14c2f6"
server_key_iv: "611fe24036eb938e849746be0d641e26"
server_key_version: "encryptionkey@0"
I20260928 21:59:04.067916 5325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:04.078482 5325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:04.081223 5325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:04.081415 5325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:04.081674 5325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:04.082015 5325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:04.082072 5325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:04.082116 5325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:04.082190 5325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:04.088574 5325 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:43937
I20260928 21:59:04.088603 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:43937 every 8 connection(s)
I20260928 21:59:04.088994 5325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:04.091279 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5325
I20260928 21:59:04.091404 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:04.091581 2709 external_mini_cluster.cc:1428] Setting key 90c81d5095c0978893a58911f73ee8dc
Sep 28 21:59:04 dist-test-slave-3t36 krb5kdc[5084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632744, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:04.100045 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:04.089745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:49607 (local address 127.2.165.126:44749)
0928 21:59:04.089817 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:04.089820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:04.090084 (+ 264us) server_negotiation.cc:407] Connection header received
0928 21:59:04.090624 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:04.090626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:04.090643 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:04.090695 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:04.091755 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.092441 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.093557 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.093771 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.093861 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:04.096483 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:04.096485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:04.096485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:04.096496 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:04.098387 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:04.098828 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:04.098830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:04.098831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:04.098898 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:04.099239 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:04.099241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:04.099241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:04.099406 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:04.099486 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:04.099837 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 21:59:04.099931 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:59:04.100194 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:04.089744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:44749 (local address 127.2.165.66:49607)
0928 21:59:04.089917 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:04.089954 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:04.090511 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:04.090772 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:04.090776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:04.091356 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:04.091636 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:04.091658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.092542 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.092543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:04.093458 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:04.093461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.093872 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.093874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:04.094038 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:04.094623 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:04.094635 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:04.096381 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:04.098483 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:04.098489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:04.098491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:04.098743 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:04.099022 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:04.099024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:04.099024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:04.099155 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:04.099482 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:04.099487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:04.099641 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 21:59:04.099781 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 21:59:04.099934 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260928 21:59:04.101049 5454 heartbeater.cc:347] Connected to a master server at 127.2.165.126:44749
I20260928 21:59:04.101169 5454 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 21:59:04.101390 5454 heartbeater.cc:511] Master 127.2.165.126:44749 requested a full tablet report, sending...
I20260928 21:59:04.101830 5129 ts_manager.cc:194] Registered new tserver with Master: 7084698f3943403fb7974c4f001009ea (127.2.165.66:43937)
I20260928 21:59:04.102202 5129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:49607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:59:04.120639 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44749
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:04.203743 5459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:04.203985 5459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:04.204013 5459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:04.205524 5459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:04.205576 5459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:04.205607 5459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:04.207191 5459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:04.207475 5459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:04.207722 5459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:04.213013 5467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:04.213073 5465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:04.213034 5464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:04.213289 5459 server_base.cc:1034] running on GCE node
I20260928 21:59:04.213522 5459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:04.213762 5459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:04.214924 5459 hybrid_clock.cc:648] HybridClock initialized: now 1790632744214906 us; error 36 us; skew 500 ppm
Sep 28 21:59:04 dist-test-slave-3t36 krb5kdc[5084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632744, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:04.217716 5459 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:59:04.218142 5459 webserver.cc:460] Webserver started at http://127.2.165.67:33777/ using document root <none> and password file <none>
I20260928 21:59:04.218341 5459 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:04.218384 5459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:04.218473 5459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:04.219667 5459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "ed61077d07c543af90a490e2b2e4166d"
format_stamp: "Formatted at 2026-09-28 21:59:04 on dist-test-slave-3t36"
server_key: "b4d219ded6180d716bc0f327adc5b1c2"
server_key_iv: "e0501f38f0951c2244d167ad14fcbe11"
server_key_version: "encryptionkey@0"
I20260928 21:59:04.220074 5459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "ed61077d07c543af90a490e2b2e4166d"
format_stamp: "Formatted at 2026-09-28 21:59:04 on dist-test-slave-3t36"
server_key: "b4d219ded6180d716bc0f327adc5b1c2"
server_key_iv: "e0501f38f0951c2244d167ad14fcbe11"
server_key_version: "encryptionkey@0"
I20260928 21:59:04.221679 5459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:04.222550 5474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:04.222784 5459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:04.222898 5459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "ed61077d07c543af90a490e2b2e4166d"
format_stamp: "Formatted at 2026-09-28 21:59:04 on dist-test-slave-3t36"
server_key: "b4d219ded6180d716bc0f327adc5b1c2"
server_key_iv: "e0501f38f0951c2244d167ad14fcbe11"
server_key_version: "encryptionkey@0"
I20260928 21:59:04.222981 5459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:04.243396 5459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:04.245970 5459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:04.246145 5459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:04.246397 5459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:04.246718 5459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:04.246771 5459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:04.246816 5459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:04.246855 5459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:04.253023 5459 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:35053
I20260928 21:59:04.253065 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:35053 every 8 connection(s)
I20260928 21:59:04.253440 5459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:04.255973 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5459
I20260928 21:59:04.256081 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:04.256244 2709 external_mini_cluster.cc:1428] Setting key 9ef833f4fc32275b41ead90d87ef9be8
Sep 28 21:59:04 dist-test-slave-3t36 krb5kdc[5084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632744, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:04.263681 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:04.254299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:40137 (local address 127.2.165.126:44749)
0928 21:59:04.254407 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:04.254409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:04.254586 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:59:04.255143 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:04.255145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:04.255159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:04.255216 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:04.256469 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.257269 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.258349 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.258562 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.258653 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:04.261068 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:04.261070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:04.261071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:04.261078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:04.262347 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:04.262773 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:04.262774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:04.262774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:04.262836 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:04.263074 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:04.263075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:04.263075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:04.263189 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:04.263286 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:04.263542 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 21:59:04.263612 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:59:04.263861 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:04.254223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:44749 (local address 127.2.165.67:40137)
0928 21:59:04.254442 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:04.254497 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:04.255018 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:04.255293 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:04.255298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:04.256055 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:04.256372 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:04.256381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.257395 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.257396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:04.258247 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:04.258249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:04.258664 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:04.258666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:04.258814 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:04.259391 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:04.259401 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:04.260971 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:04.262426 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:04.262434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:04.262436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:04.262686 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:04.262906 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:04.262906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:04.262907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:04.263017 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:04.263272 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:04.263276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:04.263408 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 21:59:04.263514 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:59:04.263644 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260928 21:59:04.264374 5588 heartbeater.cc:347] Connected to a master server at 127.2.165.126:44749
I20260928 21:59:04.264518 5588 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:04.264760 5588 heartbeater.cc:511] Master 127.2.165.126:44749 requested a full tablet report, sending...
I20260928 21:59:04.265203 5129 ts_manager.cc:194] Registered new tserver with Master: ed61077d07c543af90a490e2b2e4166d (127.2.165.67:35053)
I20260928 21:59:04.265548 5129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:40137
I20260928 21:59:04.268174 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:59:04.280056 5186 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:04.279093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.2.165.126:44749)
0928 21:59:04.279147 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:04.279149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:04.279230 (+ 81us) server_negotiation.cc:407] Connection header received
0928 21:59:04.279283 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:04.279284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:04.279295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:04.279346 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:04.279988 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47082: BlockingRecv error: recv got EOF from 127.0.0.1:47082 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:59:04.281117 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5191
I20260928 21:59:04.286003 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5325
I20260928 21:59:04.290612 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5459
I20260928 21:59:04.295094 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5097
2026-09-28T21:59:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1871 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:04 dist-test-slave-3t36 krb5kdc[5600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:04 dist-test-slave-3t36 krb5kdc[5600](info): set up 2 sockets
Sep 28 21:59:04 dist-test-slave-3t36 krb5kdc[5600](info): commencing operation
krb5kdc: starting...
W20260928 21:59:05.400580 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:05 dist-test-slave-3t36 krb5kdc[5600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:05.525115 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42511
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:05.608412 5613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:05.608688 5613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:05.608757 5613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:05.609963 5613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:05.610037 5613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:05.610061 5613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:05.610085 5613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:05.610112 5613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:05.611665 5613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:42511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:05.611943 5613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:05.612174 5613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:05.617347 5621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:05.617355 5613 server_base.cc:1034] running on GCE node
W20260928 21:59:05.617318 5619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:05.617316 5618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:05.617883 5613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:05.618211 5613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:05.619370 5613 hybrid_clock.cc:648] HybridClock initialized: now 1790632745619349 us; error 37 us; skew 500 ppm
Sep 28 21:59:05 dist-test-slave-3t36 krb5kdc[5600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632745, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:05.622141 5613 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:05.622718 5613 webserver.cc:460] Webserver started at http://127.2.165.126:36599/ using document root <none> and password file <none>
I20260928 21:59:05.622982 5613 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:05.623029 5613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:05.623193 5613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:05.624334 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "9a32b70c75764af4951f980910864cc1"
format_stamp: "Formatted at 2026-09-28 21:59:05 on dist-test-slave-3t36"
server_key: "a70564930afcfb2e4864b7ed61ae8f17"
server_key_iv: "db86b72863aecaf27120517e063d5afb"
server_key_version: "encryptionkey@0"
I20260928 21:59:05.624749 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "9a32b70c75764af4951f980910864cc1"
format_stamp: "Formatted at 2026-09-28 21:59:05 on dist-test-slave-3t36"
server_key: "a70564930afcfb2e4864b7ed61ae8f17"
server_key_iv: "db86b72863aecaf27120517e063d5afb"
server_key_version: "encryptionkey@0"
I20260928 21:59:05.626135 5613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:05.627012 5628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:05.627264 5613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:05.627391 5613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "9a32b70c75764af4951f980910864cc1"
format_stamp: "Formatted at 2026-09-28 21:59:05 on dist-test-slave-3t36"
server_key: "a70564930afcfb2e4864b7ed61ae8f17"
server_key_iv: "db86b72863aecaf27120517e063d5afb"
server_key_version: "encryptionkey@0"
I20260928 21:59:05.627485 5613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:05.649989 5613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:05.652062 5613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:05.652240 5613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:05.656622 5613 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:42511
I20260928 21:59:05.656659 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:42511 every 8 connection(s)
I20260928 21:59:05.657068 5613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:05.657979 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:05.660357 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5613
I20260928 21:59:05.660421 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1: Bootstrap starting.
I20260928 21:59:05.660471 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:05.660645 2709 external_mini_cluster.cc:1428] Setting key 8d2f4eb920d6d104624e9dc74b84a53d
I20260928 21:59:05.661192 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:05.661518 5681 log.cc:824] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:05.662271 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1: No bootstrap required, opened a new log
I20260928 21:59:05.664489 5681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a32b70c75764af4951f980910864cc1" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42511 } }
I20260928 21:59:05.664660 5681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:05.664723 5681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a32b70c75764af4951f980910864cc1, State: Initialized, Role: FOLLOWER
I20260928 21:59:05.664842 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a32b70c75764af4951f980910864cc1" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42511 } }
I20260928 21:59:05.664927 5681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:05.664968 5681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:05.665024 5681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:05.665625 5681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a32b70c75764af4951f980910864cc1" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42511 } }
I20260928 21:59:05.665738 5681 leader_election.cc:302] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a32b70c75764af4951f980910864cc1; no voters:
I20260928 21:59:05.665908 5681 leader_election.cc:288] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:05.666103 5686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:05.666198 5681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:05.666363 5686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [term 1 LEADER]: Becoming Leader. State: Replica: 9a32b70c75764af4951f980910864cc1, State: Running, Role: LEADER
I20260928 21:59:05.666464 5686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a32b70c75764af4951f980910864cc1" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42511 } }
I20260928 21:59:05.666786 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a32b70c75764af4951f980910864cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a32b70c75764af4951f980910864cc1" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42511 } } }
I20260928 21:59:05.666880 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:05.666771 5686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a32b70c75764af4951f980910864cc1. Latest consensus state: current_term: 1 leader_uuid: "9a32b70c75764af4951f980910864cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a32b70c75764af4951f980910864cc1" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42511 } } }
I20260928 21:59:05.667068 5686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:05.667174 5695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:05.667681 5695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:59:05 dist-test-slave-3t36 krb5kdc[5600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:05.669910 5695 catalog_manager.cc:1347] Generated new cluster ID: eca9d805bc0d49eda37b63ac5fffe10b
I20260928 21:59:05.669972 5695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:05.682900 5695 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:05.683656 5695 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:05.683801 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:05.661216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.2.165.126:42511)
0928 21:59:05.661419 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:05.661436 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:05.661462 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:59:05.661989 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:05.661997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:05.662050 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:05.662174 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:05.663025 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:05.663934 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:05.665057 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:05.665506 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:05.665599 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:05.668765 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:05.668770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:05.668770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:05.668783 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:05.681565 (+ 12782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:05.682095 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:05.682101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:05.682102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:05.682248 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:05.682690 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:05.682691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:05.682692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:05.683003 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:05.683191 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:05.683354 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:59:05.683515 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260928 21:59:05.684798 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:59:05.689218 5695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a32b70c75764af4951f980910864cc1: Generated new TSK 0
I20260928 21:59:05.689420 5695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:59:05.709125 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:05.697603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.2.165.126:42511)
0928 21:59:05.697673 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:05.697676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:05.697887 (+ 211us) server_negotiation.cc:407] Connection header received
0928 21:59:05.697923 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:05.697924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:05.697934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:05.697991 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:05.698532 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:05.699126 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:05.700241 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:05.700466 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:05.700532 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:05.708400 (+ 7868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:05.708402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:05.708402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:05.708606 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:05.708628 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:05.708635 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:05.708840 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:05.708980 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:59:05.709016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:59:05.712530 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5613
2026-09-28T21:59:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1415 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:05 dist-test-slave-3t36 krb5kdc[5714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:05 dist-test-slave-3t36 krb5kdc[5714](info): set up 2 sockets
Sep 28 21:59:05 dist-test-slave-3t36 krb5kdc[5714](info): commencing operation
krb5kdc: starting...
W20260928 21:59:06.768622 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:06 dist-test-slave-3t36 krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:06.895818 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42333
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:42333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:06.981050 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:06.981283 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:06.981308 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:06.982512 5727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:06.982558 5727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:06.982573 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:06.982585 5727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:06.982599 5727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:06.984115 5727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:42333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:06.984336 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:06.984530 5727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:06.989773 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:06.989773 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:06.989768 5732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:06.990195 5727 server_base.cc:1034] running on GCE node
I20260928 21:59:06.990456 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:06.990787 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:06.991976 5727 hybrid_clock.cc:648] HybridClock initialized: now 1790632746991937 us; error 58 us; skew 500 ppm
Sep 28 21:59:06 dist-test-slave-3t36 krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632746, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:06.994885 5727 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:06.995402 5727 webserver.cc:460] Webserver started at http://127.2.165.126:39877/ using document root <none> and password file <none>
I20260928 21:59:06.995658 5727 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:06.995728 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:06.995888 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:06.997063 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "abed1f1c16054c6ca005e4180c269c6e"
format_stamp: "Formatted at 2026-09-28 21:59:06 on dist-test-slave-3t36"
server_key: "45607bb88d9e085d884c08a171bd246c"
server_key_iv: "787ece7bd290522ffbec60a7ae768cff"
server_key_version: "encryptionkey@0"
I20260928 21:59:06.997440 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "abed1f1c16054c6ca005e4180c269c6e"
format_stamp: "Formatted at 2026-09-28 21:59:06 on dist-test-slave-3t36"
server_key: "45607bb88d9e085d884c08a171bd246c"
server_key_iv: "787ece7bd290522ffbec60a7ae768cff"
server_key_version: "encryptionkey@0"
I20260928 21:59:06.998939 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:06.999831 5742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.000108 5727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:07.000236 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "abed1f1c16054c6ca005e4180c269c6e"
format_stamp: "Formatted at 2026-09-28 21:59:06 on dist-test-slave-3t36"
server_key: "45607bb88d9e085d884c08a171bd246c"
server_key_iv: "787ece7bd290522ffbec60a7ae768cff"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.000339 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:07.013523 5727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:07.015564 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:07.015736 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:07.019872 5727 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:42333
I20260928 21:59:07.019896 5794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:42333 every 8 connection(s)
I20260928 21:59:07.020259 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:07.021531 5795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:07.021580 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5727
I20260928 21:59:07.021670 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:07.021864 2709 external_mini_cluster.cc:1428] Setting key 6f4a5192a7b42277a266228b5b970e46
I20260928 21:59:07.024991 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e: Bootstrap starting.
I20260928 21:59:07.025763 5795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:07.026085 5795 log.cc:824] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:07.026841 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e: No bootstrap required, opened a new log
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:07.029518 5795 raft_consensus.cc:348] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abed1f1c16054c6ca005e4180c269c6e" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42333 } }
I20260928 21:59:07.029667 5795 raft_consensus.cc:374] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:07.029691 5795 raft_consensus.cc:729] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abed1f1c16054c6ca005e4180c269c6e, State: Initialized, Role: FOLLOWER
I20260928 21:59:07.029862 5795 consensus_queue.cc:260] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abed1f1c16054c6ca005e4180c269c6e" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42333 } }
I20260928 21:59:07.029969 5795 raft_consensus.cc:388] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:07.030009 5795 raft_consensus.cc:482] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:07.030063 5795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:07.030674 5795 raft_consensus.cc:504] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abed1f1c16054c6ca005e4180c269c6e" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42333 } }
I20260928 21:59:07.030804 5795 leader_election.cc:302] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abed1f1c16054c6ca005e4180c269c6e; no voters:
I20260928 21:59:07.031028 5795 leader_election.cc:288] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:07.031056 5800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:07.031167 5800 raft_consensus.cc:686] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [term 1 LEADER]: Becoming Leader. State: Replica: abed1f1c16054c6ca005e4180c269c6e, State: Running, Role: LEADER
I20260928 21:59:07.031251 5800 consensus_queue.cc:237] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abed1f1c16054c6ca005e4180c269c6e" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42333 } }
I20260928 21:59:07.031322 5795 sys_catalog.cc:564] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:07.031584 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abed1f1c16054c6ca005e4180c269c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abed1f1c16054c6ca005e4180c269c6e" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42333 } } }
I20260928 21:59:07.031706 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:07.031870 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader abed1f1c16054c6ca005e4180c269c6e. Latest consensus state: current_term: 1 leader_uuid: "abed1f1c16054c6ca005e4180c269c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abed1f1c16054c6ca005e4180c269c6e" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42333 } } }
I20260928 21:59:07.032011 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:07.032516 5805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:07.033062 5805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:07.033926 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.022413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.2.165.126:42333)
0928 21:59:07.022571 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:07.022579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:07.022608 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:59:07.023091 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:07.023099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:07.023151 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:07.023286 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:07.024139 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.025181 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.026306 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.026833 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.026919 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.029624 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:07.029629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:07.029629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:07.029641 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:07.031905 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.032335 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.032338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.032338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.032450 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.032647 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.032648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.032648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.033026 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:07.033186 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:07.033588 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 21:59:07.033729 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260928 21:59:07.038498 5805 catalog_manager.cc:1347] Generated new cluster ID: 3490a358f6a048289c332214c0630757
I20260928 21:59:07.038573 5805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:07.053817 5805 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:07.054615 5805 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:07.069036 5805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abed1f1c16054c6ca005e4180c269c6e: Generated new TSK 0
I20260928 21:59:07.069286 5805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:59:07.100620 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42333
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:07.190739 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:07.191069 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:07.191114 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:07.192508 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:07.192561 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:07.192593 5821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:07.194234 5821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:07.194463 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:07.194661 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:07.200135 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:07.200094 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:07.200094 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:07.200472 5821 server_base.cc:1034] running on GCE node
I20260928 21:59:07.200802 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:07.201092 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:07.202288 5821 hybrid_clock.cc:648] HybridClock initialized: now 1790632747202278 us; error 43 us; skew 500 ppm
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:07.205215 5821 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:59:07.205633 5821 webserver.cc:460] Webserver started at http://127.2.165.65:43691/ using document root <none> and password file <none>
I20260928 21:59:07.205873 5821 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:07.205919 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:07.206014 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:07.207064 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "5fa1a0716c50493e80431f75111494eb"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "1c81d13c87eea59d71daa9808545998a"
server_key_iv: "2f115327a0020bdcf705e9d694c2e6c3"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.207377 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "5fa1a0716c50493e80431f75111494eb"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "1c81d13c87eea59d71daa9808545998a"
server_key_iv: "2f115327a0020bdcf705e9d694c2e6c3"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.208758 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:07.209631 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.209794 5821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:07.209879 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "5fa1a0716c50493e80431f75111494eb"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "1c81d13c87eea59d71daa9808545998a"
server_key_iv: "2f115327a0020bdcf705e9d694c2e6c3"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.209940 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:07.227311 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:07.230111 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:07.230234 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:07.230430 5821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:07.230829 5821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:07.230867 5821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.230892 5821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:07.230906 5821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.236713 5821 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:39141
I20260928 21:59:07.236783 5949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:39141 every 8 connection(s)
I20260928 21:59:07.237149 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:07.246246 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5821
I20260928 21:59:07.246363 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:07.246594 2709 external_mini_cluster.cc:1428] Setting key 36abfb16adc48fb75bf083aaaf6fb3a0
I20260928 21:59:07.247505 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.237999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:38131 (local address 127.2.165.126:42333)
0928 21:59:07.238091 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:07.238093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:07.238282 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:59:07.238776 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:07.238778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:07.238789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:07.238846 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:07.239783 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.240316 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.241449 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.241680 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.241751 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.244104 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:07.244106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:07.244106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:07.244114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:07.245794 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.246223 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.246225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.246226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.246290 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.246651 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.246652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.246653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.246808 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:07.246900 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:07.247310 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 21:59:07.247405 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:59:07.247743 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.237963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42333 (local address 127.2.165.65:38131)
0928 21:59:07.238156 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:07.238194 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:07.238677 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:07.238900 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:07.238904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:07.239475 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:07.239680 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:07.239684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.240407 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.240407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:07.241329 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:07.241332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.241764 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.241766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:07.241869 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.242410 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:07.242419 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:07.243989 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:07.245879 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:07.245887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:07.245889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:07.246140 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:07.246431 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:07.246433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:07.246433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:07.246575 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:07.246961 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:07.246968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:07.247145 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 21:59:07.247289 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 21:59:07.247462 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260928 21:59:07.248517 5950 heartbeater.cc:347] Connected to a master server at 127.2.165.126:42333
I20260928 21:59:07.248632 5950 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:07.248870 5950 heartbeater.cc:511] Master 127.2.165.126:42333 requested a full tablet report, sending...
I20260928 21:59:07.249511 5759 ts_manager.cc:194] Registered new tserver with Master: 5fa1a0716c50493e80431f75111494eb (127.2.165.65:39141)
I20260928 21:59:07.250144 5759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:38131
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:59:07.274123 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42333
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:07.358192 5955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:07.358440 5955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:07.358469 5955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:07.359773 5955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:07.359833 5955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:07.359865 5955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:07.361543 5955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:07.361833 5955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:07.362082 5955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:59:07.367336 5955 server_base.cc:1034] running on GCE node
W20260928 21:59:07.367349 5963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:07.367336 5961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:07.367349 5960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:07.367782 5955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:07.368005 5955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:07.369174 5955 hybrid_clock.cc:648] HybridClock initialized: now 1790632747369161 us; error 31 us; skew 500 ppm
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:07.371814 5955 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:59:07.372321 5955 webserver.cc:460] Webserver started at http://127.2.165.66:38019/ using document root <none> and password file <none>
I20260928 21:59:07.372572 5955 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:07.372618 5955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:07.372812 5955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:07.373914 5955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "e6efbdda2f9c438f8f8a55d9c13a035b"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "7647b9ea0176b03fb48358b310cb5e2a"
server_key_iv: "7b26c26b1dac739ae2659f1f2654db01"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.374301 5955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "e6efbdda2f9c438f8f8a55d9c13a035b"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "7647b9ea0176b03fb48358b310cb5e2a"
server_key_iv: "7b26c26b1dac739ae2659f1f2654db01"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.375743 5955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:07.376746 5970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.376993 5955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:07.377118 5955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "e6efbdda2f9c438f8f8a55d9c13a035b"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "7647b9ea0176b03fb48358b310cb5e2a"
server_key_iv: "7b26c26b1dac739ae2659f1f2654db01"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.377205 5955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:07.390756 5955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:07.393157 5955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:07.393335 5955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:07.393585 5955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:07.393922 5955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:07.393978 5955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.394085 5955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:07.394120 5955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.400240 5955 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:36087
I20260928 21:59:07.400280 6083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:36087 every 8 connection(s)
I20260928 21:59:07.400624 5955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:07.409512 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 5955
I20260928 21:59:07.409634 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:07.409829 2709 external_mini_cluster.cc:1428] Setting key 5c6d93c02b5c9a159ea972993ae17400
I20260928 21:59:07.411227 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.401411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42333 (local address 127.2.165.66:42485)
0928 21:59:07.401609 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:07.401687 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:07.402319 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:07.402616 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:07.402620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:07.403262 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:07.403467 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:07.403472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.404229 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.404231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:07.405147 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:07.405150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.405554 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.405555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:07.405689 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.406278 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:07.406287 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:07.407749 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:07.409514 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:07.409522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:07.409525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:07.409812 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:07.410035 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:07.410036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:07.410036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:07.410166 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:07.410604 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:07.410612 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:07.410757 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 21:59:07.410880 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:59:07.411025 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260928 21:59:07.411244 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.401439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:42485 (local address 127.2.165.126:42333)
0928 21:59:07.401547 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:07.401550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:07.401798 (+ 248us) server_negotiation.cc:407] Connection header received
0928 21:59:07.402440 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:07.402442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:07.402455 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:07.402522 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:07.403570 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.404124 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.405243 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.405452 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.405544 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.407856 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:07.407858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:07.407859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:07.407868 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:07.409404 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.409903 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.409905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.409906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.409964 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.410275 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.410277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.410277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.410516 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:07.410594 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:07.410931 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 21:59:07.411103 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:59:07.411891 6084 heartbeater.cc:347] Connected to a master server at 127.2.165.126:42333
I20260928 21:59:07.412065 6084 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:07.412261 6084 heartbeater.cc:511] Master 127.2.165.126:42333 requested a full tablet report, sending...
I20260928 21:59:07.412739 5759 ts_manager.cc:194] Registered new tserver with Master: e6efbdda2f9c438f8f8a55d9c13a035b (127.2.165.66:36087)
I20260928 21:59:07.413190 5759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:42485
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:59:07.436049 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42333
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:07.520169 6089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:07.520411 6089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:07.520452 6089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:07.521860 6089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:07.521934 6089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:07.521977 6089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:07.523474 6089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:07.523765 6089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:07.524036 6089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:07.529405 6095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:07.529335 6094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:07.529335 6097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:07.529575 6089 server_base.cc:1034] running on GCE node
I20260928 21:59:07.529899 6089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:07.530174 6089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:07.531327 6089 hybrid_clock.cc:648] HybridClock initialized: now 1790632747531316 us; error 32 us; skew 500 ppm
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:07.534338 6089 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:59:07.534829 6089 webserver.cc:460] Webserver started at http://127.2.165.67:41155/ using document root <none> and password file <none>
I20260928 21:59:07.535065 6089 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:07.535111 6089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:07.535241 6089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:07.536329 6089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "74762832353b48cb8c187ec13001b573"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "055063aaf89d33d67d5de45a80428f33"
server_key_iv: "5cdd26a6a92930ed94d36a35fcbc5dec"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.536758 6089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "74762832353b48cb8c187ec13001b573"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "055063aaf89d33d67d5de45a80428f33"
server_key_iv: "5cdd26a6a92930ed94d36a35fcbc5dec"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.538148 6089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:07.538914 6104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.539120 6089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:59:07.539211 6089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "74762832353b48cb8c187ec13001b573"
format_stamp: "Formatted at 2026-09-28 21:59:07 on dist-test-slave-3t36"
server_key: "055063aaf89d33d67d5de45a80428f33"
server_key_iv: "5cdd26a6a92930ed94d36a35fcbc5dec"
server_key_version: "encryptionkey@0"
I20260928 21:59:07.539290 6089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:07.554705 6089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:07.557164 6089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:07.557329 6089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:07.557598 6089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:07.557948 6089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:07.558012 6089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.558056 6089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:07.558084 6089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:07.564054 6089 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:46355
I20260928 21:59:07.564129 6217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:46355 every 8 connection(s)
I20260928 21:59:07.564495 6089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:07.571421 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6089
I20260928 21:59:07.571528 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:07.571707 2709 external_mini_cluster.cc:1428] Setting key 2f7a4980d2b719fc5777ce70aa68a519
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:07.575659 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.565482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:53179 (local address 127.2.165.126:42333)
0928 21:59:07.565529 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:07.565531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:07.565704 (+ 173us) server_negotiation.cc:407] Connection header received
0928 21:59:07.566383 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:07.566385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:07.566397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:07.566463 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:07.567494 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.568040 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.569588 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.569819 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.569885 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.572771 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:07.572773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:07.572773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:07.572785 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:07.574185 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.574627 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.574628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.574629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.574690 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.574961 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.574962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.574962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.575093 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:07.575175 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:07.575465 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 21:59:07.575554 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:59:07.575902 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.565385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42333 (local address 127.2.165.67:53179)
0928 21:59:07.565543 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:07.565586 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:07.566233 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:07.566523 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:07.566526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:07.567131 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:07.567341 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:07.567346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.568167 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.568168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:07.569465 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:07.569468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.569955 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.569956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:07.570084 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.570786 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:07.570796 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:07.572650 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:07.574283 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:07.574292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:07.574295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:07.574546 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:07.574760 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:07.574761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:07.574762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:07.574891 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:07.575163 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:07.575170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:07.575332 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 21:59:07.575445 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 21:59:07.575635 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260928 21:59:07.576409 6218 heartbeater.cc:347] Connected to a master server at 127.2.165.126:42333
I20260928 21:59:07.576507 6218 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:07.576663 6218 heartbeater.cc:511] Master 127.2.165.126:42333 requested a full tablet report, sending...
I20260928 21:59:07.577114 5759 ts_manager.cc:194] Registered new tserver with Master: 74762832353b48cb8c187ec13001b573 (127.2.165.67:46355)
I20260928 21:59:07.577519 5759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:53179
I20260928 21:59:07.583940 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[5714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632747, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:07.609254 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:07.600766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.2.165.126:42333)
0928 21:59:07.600833 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:07.600835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:07.600910 (+ 75us) server_negotiation.cc:407] Connection header received
0928 21:59:07.600961 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:07.600962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:07.600972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:07.601075 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:07.601682 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.602182 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.603072 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:07.603270 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:07.603325 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:07.605992 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:07.605994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:07.605994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:07.606002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:07.607702 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.608171 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.608172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.608172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.608231 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:07.608498 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:07.608500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:07.608500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:07.608766 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:07.608857 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:07.609059 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 21:59:07.609182 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 21:59:07.609586 5759 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49326
I20260928 21:59:07.610252 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5821
I20260928 21:59:07.615193 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5955
I20260928 21:59:07.619843 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6089
I20260928 21:59:07.624825 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 5727
2026-09-28T21:59:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1914 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[6230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[6230](info): set up 2 sockets
Sep 28 21:59:07 dist-test-slave-3t36 krb5kdc[6230](info): commencing operation
krb5kdc: starting...
W20260928 21:59:08.683625 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 21:59:08.820612 2709 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.137s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 21:59:09.041553 2709 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.221s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 21:59:09.309648 2709 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.268s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/test-user.keytab.
W20260928 21:59:10.298060 2709 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.988s user 0.000s sys 0.001s
Sep 28 21:59:10 dist-test-slave-3t36 krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
W20260928 21:59:11.537509 2709 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.165.126: real 1.217s user 0.001s sys 0.002s
I20260928 21:59:11.537932 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:36041
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:11.620246 6243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:11.620482 6243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:11.620505 6243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:11.621695 6243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:11.621737 6243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:11.621752 6243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:11.621763 6243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:11.621775 6243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:11.623248 6243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:11.623456 6243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:11.623629 6243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:11.628765 6248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:11.628752 6249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:11.628752 6251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:11.628952 6243 server_base.cc:1034] running on GCE node
I20260928 21:59:11.629315 6243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:11.629647 6243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:11.630813 6243 hybrid_clock.cc:648] HybridClock initialized: now 1790632751630796 us; error 42 us; skew 500 ppm
Sep 28 21:59:11 dist-test-slave-3t36 krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632751, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:11.641759 6243 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:11.642284 6243 webserver.cc:460] Webserver started at http://127.2.165.126:38189/ using document root <none> and password file <none>
I20260928 21:59:11.642546 6243 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:11.642623 6243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:11.642797 6243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:11.644030 6243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "a8986b37a99146a88e09bd8fa5ddcd40"
format_stamp: "Formatted at 2026-09-28 21:59:11 on dist-test-slave-3t36"
server_key: "757fc3c66ea388b98ddabe34336aa333"
server_key_iv: "4e7fb742ebaa0f35e8e21fe8f2ac3727"
server_key_version: "encryptionkey@0"
I20260928 21:59:11.644433 6243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "a8986b37a99146a88e09bd8fa5ddcd40"
format_stamp: "Formatted at 2026-09-28 21:59:11 on dist-test-slave-3t36"
server_key: "757fc3c66ea388b98ddabe34336aa333"
server_key_iv: "4e7fb742ebaa0f35e8e21fe8f2ac3727"
server_key_version: "encryptionkey@0"
I20260928 21:59:11.645912 6243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:11.646831 6258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:11.647143 6243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:11.647265 6243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "a8986b37a99146a88e09bd8fa5ddcd40"
format_stamp: "Formatted at 2026-09-28 21:59:11 on dist-test-slave-3t36"
server_key: "757fc3c66ea388b98ddabe34336aa333"
server_key_iv: "4e7fb742ebaa0f35e8e21fe8f2ac3727"
server_key_version: "encryptionkey@0"
I20260928 21:59:11.647347 6243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:11.664855 6243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:11.667248 6243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:11.667441 6243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:11.671840 6243 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:36041
I20260928 21:59:11.671898 6310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:36041 every 8 connection(s)
I20260928 21:59:11.672492 6243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:11.673375 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:11.673360 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6243
I20260928 21:59:11.673475 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:11.673669 2709 external_mini_cluster.cc:1428] Setting key 5f55e9ec4489a293a7f0941e19408919
I20260928 21:59:11.676905 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40: Bootstrap starting.
I20260928 21:59:11.677714 6311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:11.678025 6311 log.cc:824] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:11.678785 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40: No bootstrap required, opened a new log
Sep 28 21:59:11 dist-test-slave-3t36 krb5kdc[6230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:11.681394 6311 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36041 } }
I20260928 21:59:11.681538 6311 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:11.681565 6311 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8986b37a99146a88e09bd8fa5ddcd40, State: Initialized, Role: FOLLOWER
I20260928 21:59:11.681741 6311 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36041 } }
I20260928 21:59:11.681821 6311 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:11.681844 6311 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:11.681879 6311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:11.684990 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:11.674312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.2.165.126:36041)
0928 21:59:11.674485 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:11.674494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:11.674532 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:59:11.675006 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:11.675014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:11.675072 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:11.675197 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:11.676038 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:11.677042 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:11.678136 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:11.678627 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:11.678721 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:11.681462 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:11.681466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:11.681466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:11.681478 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:11.683381 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:11.683779 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:11.683782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:11.683783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:11.683872 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:11.684079 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:11.684080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:11.684080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:11.684334 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:11.684488 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:11.684606 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 21:59:11.684775 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 21:59:11.687538 6311 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36041 } }
I20260928 21:59:11.687700 6311 leader_election.cc:302] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8986b37a99146a88e09bd8fa5ddcd40; no voters:
I20260928 21:59:11.687919 6311 leader_election.cc:288] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:11.688105 6317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:11.688256 6311 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:11.688329 6317 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [term 1 LEADER]: Becoming Leader. State: Replica: a8986b37a99146a88e09bd8fa5ddcd40, State: Running, Role: LEADER
I20260928 21:59:11.688431 6317 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36041 } }
I20260928 21:59:11.688858 6317 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8986b37a99146a88e09bd8fa5ddcd40. Latest consensus state: current_term: 1 leader_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36041 } } }
I20260928 21:59:11.689006 6317 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:11.689248 6325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:11.689360 6316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8986b37a99146a88e09bd8fa5ddcd40" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36041 } } }
I20260928 21:59:11.689544 6316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:11.689788 6325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:11.693665 6325 catalog_manager.cc:1347] Generated new cluster ID: cad8e12a4ee64483ad465db693ec6937
I20260928 21:59:11.693754 6325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:11.703760 6325 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:11.704535 6325 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:11.708966 6325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8986b37a99146a88e09bd8fa5ddcd40: Generated new TSK 0
I20260928 21:59:11.709190 6325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
W20260928 21:59:12.837285 2709 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.165.65: real 1.121s user 0.000s sys 0.003s
I20260928 21:59:12.837729 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:36041
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:12.921552 6337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:12.921785 6337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:12.921811 6337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:12.923092 6337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:12.923156 6337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:12.923192 6337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:12.924889 6337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:12.925165 6337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:12.925451 6337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:12.931041 6342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:12.931051 6345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:12.931098 6343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:12.931169 6337 server_base.cc:1034] running on GCE node
I20260928 21:59:12.931625 6337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:12.931907 6337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:12.933122 6337 hybrid_clock.cc:648] HybridClock initialized: now 1790632752933107 us; error 28 us; skew 500 ppm
Sep 28 21:59:12 dist-test-slave-3t36 krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:12.936002 6337 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:59:12.936532 6337 webserver.cc:460] Webserver started at http://127.2.165.65:34293/ using document root <none> and password file <none>
I20260928 21:59:12.936846 6337 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:12.936916 6337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:12.937090 6337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:12.938213 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "33fc6ec390e14821979349d8f96e4c7c"
format_stamp: "Formatted at 2026-09-28 21:59:12 on dist-test-slave-3t36"
server_key: "dce967a4a383ef28151e08ff7d42a699"
server_key_iv: "5b3464e76b7d37b918437c2d49427165"
server_key_version: "encryptionkey@0"
I20260928 21:59:12.938596 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "33fc6ec390e14821979349d8f96e4c7c"
format_stamp: "Formatted at 2026-09-28 21:59:12 on dist-test-slave-3t36"
server_key: "dce967a4a383ef28151e08ff7d42a699"
server_key_iv: "5b3464e76b7d37b918437c2d49427165"
server_key_version: "encryptionkey@0"
I20260928 21:59:12.940042 6337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:12.941037 6352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:12.941262 6337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:12.941372 6337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "33fc6ec390e14821979349d8f96e4c7c"
format_stamp: "Formatted at 2026-09-28 21:59:12 on dist-test-slave-3t36"
server_key: "dce967a4a383ef28151e08ff7d42a699"
server_key_iv: "5b3464e76b7d37b918437c2d49427165"
server_key_version: "encryptionkey@0"
I20260928 21:59:12.941452 6337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:12.956063 6337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:12.958634 6337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:12.958868 6337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:12.959153 6337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:12.959486 6337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:12.959542 6337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:12.959587 6337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:12.959610 6337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:12.965731 6337 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:44319
I20260928 21:59:12.965814 6465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:44319 every 8 connection(s)
I20260928 21:59:12.966168 6337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:59:12 dist-test-slave-3t36 krb5kdc[6230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632752, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:12.973249 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6337
I20260928 21:59:12.973361 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:12.973553 2709 external_mini_cluster.cc:1428] Setting key f6c34d8e89a9c5023f3422d557688cb3
I20260928 21:59:12.976660 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:12.966990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:55515 (local address 127.2.165.126:36041)
0928 21:59:12.967110 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:12.967112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:12.967339 (+ 227us) server_negotiation.cc:407] Connection header received
0928 21:59:12.968286 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:12.968287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:12.968301 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:12.968378 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:12.969045 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:12.969572 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:12.970600 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:12.970790 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:12.970843 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:12.973317 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:12.973318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:12.973318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:12.973325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:12.975041 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:12.975581 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:12.975583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:12.975583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:12.975660 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:12.975934 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:12.975935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:12.975935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:12.976090 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:12.976163 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:12.976438 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:59:12.976541 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260928 21:59:12.976804 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:12.966933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:36041 (local address 127.2.165.65:55515)
0928 21:59:12.967181 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:12.967222 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:12.968137 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:12.968445 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:12.968449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:12.968739 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:12.968950 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:12.968954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:12.969674 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:12.969675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:12.970511 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:12.970513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:12.970869 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:12.970870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:12.970976 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:12.971589 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:12.971598 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:12.973179 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:12.975131 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:12.975139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:12.975142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:12.975491 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:12.975742 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:12.975742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:12.975743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:12.975864 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:12.976164 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:12.976169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:12.976300 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 21:59:12.976392 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:59:12.976531 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 21:59:12.977509 6466 heartbeater.cc:347] Connected to a master server at 127.2.165.126:36041
I20260928 21:59:12.977613 6466 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:12.977890 6466 heartbeater.cc:511] Master 127.2.165.126:36041 requested a full tablet report, sending...
I20260928 21:59:12.978552 6275 ts_manager.cc:194] Registered new tserver with Master: 33fc6ec390e14821979349d8f96e4c7c (127.2.165.65:44319)
I20260928 21:59:12.979166 6275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:55515
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:59:13.001683 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:36041
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:13.085333 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:13.085623 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:13.085666 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:13.086900 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:13.086979 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:13.087033 6472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:13.088850 6472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:13.089143 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:13.089389 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:13.094826 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:13.094950 6472 server_base.cc:1034] running on GCE node
W20260928 21:59:13.094838 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:13.094831 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:13.095319 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:13.095573 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:13.096745 6472 hybrid_clock.cc:648] HybridClock initialized: now 1790632753096724 us; error 31 us; skew 500 ppm
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:13.099356 6472 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:59:13.099819 6472 webserver.cc:460] Webserver started at http://127.2.165.66:45763/ using document root <none> and password file <none>
I20260928 21:59:13.100076 6472 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:13.100145 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:13.100311 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:13.101428 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "c5707875c7ec4e54b96ecace116770d9"
format_stamp: "Formatted at 2026-09-28 21:59:13 on dist-test-slave-3t36"
server_key: "bfb52b314e826ed01035f6c0d32ab6f4"
server_key_iv: "5afc62f508a2475ac14ffbe8cf4fd580"
server_key_version: "encryptionkey@0"
I20260928 21:59:13.101807 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "c5707875c7ec4e54b96ecace116770d9"
format_stamp: "Formatted at 2026-09-28 21:59:13 on dist-test-slave-3t36"
server_key: "bfb52b314e826ed01035f6c0d32ab6f4"
server_key_iv: "5afc62f508a2475ac14ffbe8cf4fd580"
server_key_version: "encryptionkey@0"
I20260928 21:59:13.103163 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:13.103933 6487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:13.104182 6472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:13.104319 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "c5707875c7ec4e54b96ecace116770d9"
format_stamp: "Formatted at 2026-09-28 21:59:13 on dist-test-slave-3t36"
server_key: "bfb52b314e826ed01035f6c0d32ab6f4"
server_key_iv: "5afc62f508a2475ac14ffbe8cf4fd580"
server_key_version: "encryptionkey@0"
I20260928 21:59:13.104411 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:13.123003 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:13.125284 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:13.125452 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:13.125694 6472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:13.126019 6472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:13.126076 6472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:13.126120 6472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:13.126148 6472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:13.131923 6472 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:43963
I20260928 21:59:13.131968 6600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:43963 every 8 connection(s)
I20260928 21:59:13.132309 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:13.136914 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6472
I20260928 21:59:13.137046 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:13.137256 2709 external_mini_cluster.cc:1428] Setting key 959f011b64a844fa3a1fdceaf9009cde
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:13.143362 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:13.133229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:33657 (local address 127.2.165.126:36041)
0928 21:59:13.133310 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:13.133323 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:13.133411 (+ 88us) server_negotiation.cc:407] Connection header received
0928 21:59:13.134424 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:13.134425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:13.134438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:13.134493 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:13.135109 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.135603 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.136764 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.136998 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.137109 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:13.139726 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:13.139728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:13.139728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:13.139737 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:13.141705 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.142245 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.142247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.142247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.142325 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.142580 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.142581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.142581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.142744 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:13.142824 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:13.143084 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 21:59:13.143205 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:59:13.143375 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:13.133061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:36041 (local address 127.2.165.66:33657)
0928 21:59:13.133253 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:13.133318 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:13.134322 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:13.134551 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:13.134554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:13.134823 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:13.135018 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:13.135022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.135706 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.135707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:13.136565 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:13.136568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.137124 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.137125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:13.137284 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:13.137929 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:13.137940 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:13.139626 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:13.141805 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:13.141812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:13.141814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:13.142156 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:13.142398 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:13.142399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:13.142399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:13.142508 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:13.142815 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:13.142819 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:13.142940 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 21:59:13.143027 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:59:13.143157 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 21:59:13.144054 6601 heartbeater.cc:347] Connected to a master server at 127.2.165.126:36041
I20260928 21:59:13.144183 6601 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:13.144449 6601 heartbeater.cc:511] Master 127.2.165.126:36041 requested a full tablet report, sending...
I20260928 21:59:13.144893 6275 ts_manager.cc:194] Registered new tserver with Master: c5707875c7ec4e54b96ecace116770d9 (127.2.165.66:43963)
I20260928 21:59:13.145386 6275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:33657
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:59:13.167903 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:36041
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:13.252053 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:13.252282 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:13.252310 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:13.253621 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:13.253685 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:13.253717 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:13.255321 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:13.255570 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:13.255774 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:13.260967 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:13.261025 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:13.261025 6614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:13.261287 6606 server_base.cc:1034] running on GCE node
I20260928 21:59:13.261509 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:13.261763 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:13.262948 6606 hybrid_clock.cc:648] HybridClock initialized: now 1790632753262939 us; error 37 us; skew 500 ppm
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:13.265627 6606 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:59:13.266149 6606 webserver.cc:460] Webserver started at http://127.2.165.67:34909/ using document root <none> and password file <none>
I20260928 21:59:13.266415 6606 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:13.266486 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:13.266636 6606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:13.267685 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "c5eabbb2e1b248d7848fcda31392698f"
format_stamp: "Formatted at 2026-09-28 21:59:13 on dist-test-slave-3t36"
server_key: "8fc4eb133a0663ea3748f39594c3f82f"
server_key_iv: "91894a9f37bc4f090b27d3554a4f7f3d"
server_key_version: "encryptionkey@0"
I20260928 21:59:13.268056 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "c5eabbb2e1b248d7848fcda31392698f"
format_stamp: "Formatted at 2026-09-28 21:59:13 on dist-test-slave-3t36"
server_key: "8fc4eb133a0663ea3748f39594c3f82f"
server_key_iv: "91894a9f37bc4f090b27d3554a4f7f3d"
server_key_version: "encryptionkey@0"
I20260928 21:59:13.269497 6606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:13.270272 6621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:13.270450 6606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:13.270581 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "c5eabbb2e1b248d7848fcda31392698f"
format_stamp: "Formatted at 2026-09-28 21:59:13 on dist-test-slave-3t36"
server_key: "8fc4eb133a0663ea3748f39594c3f82f"
server_key_iv: "91894a9f37bc4f090b27d3554a4f7f3d"
server_key_version: "encryptionkey@0"
I20260928 21:59:13.270661 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:13.286182 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:13.288335 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:13.288506 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:13.288781 6606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:13.289100 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:13.289155 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:13.289230 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:13.289264 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:13.295621 6606 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:46701
I20260928 21:59:13.295702 6734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:46701 every 8 connection(s)
I20260928 21:59:13.296065 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632753, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:13.303517 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6606
I20260928 21:59:13.303663 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:13.303879 2709 external_mini_cluster.cc:1428] Setting key a5eec139102c49c01d62d9bfbee9d205
I20260928 21:59:13.307480 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:13.297056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:42591 (local address 127.2.165.126:36041)
0928 21:59:13.297131 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:13.297133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:13.297424 (+ 291us) server_negotiation.cc:407] Connection header received
0928 21:59:13.298459 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:13.298460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:13.298471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:13.298539 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:13.299321 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.299855 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.301063 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.301325 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.301394 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:13.303903 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:13.303905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:13.303905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:13.303916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:13.305757 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.306201 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.306202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.306202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.306261 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.306566 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.306568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.306568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.306792 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:13.306879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:13.307224 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 21:59:13.307389 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:59:13.307590 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:13.297011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:36041 (local address 127.2.165.67:42591)
0928 21:59:13.297295 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:13.297334 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:13.298347 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:13.298613 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:13.298617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:13.298879 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:13.299080 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:13.299083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.299969 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.299970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:13.300969 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:13.300972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.301430 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.301432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:13.301595 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:13.302140 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:13.302149 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:13.303789 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:13.305851 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:13.305859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:13.305861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:13.306108 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:13.306347 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:13.306348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:13.306348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:13.306469 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:13.306878 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:13.306883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:13.307015 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 21:59:13.307151 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 21:59:13.307366 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":189,"threads_started":1}
I20260928 21:59:13.308144 6735 heartbeater.cc:347] Connected to a master server at 127.2.165.126:36041
I20260928 21:59:13.308264 6735 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:13.308493 6735 heartbeater.cc:511] Master 127.2.165.126:36041 requested a full tablet report, sending...
I20260928 21:59:13.309080 6275 ts_manager.cc:194] Registered new tserver with Master: c5eabbb2e1b248d7848fcda31392698f (127.2.165.67:46701)
I20260928 21:59:13.309497 6275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:42591
I20260928 21:59:13.316043 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:59:13.340644 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:13.332409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.2.165.65:44319)
0928 21:59:13.332576 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:13.332586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:13.332597 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:13.332625 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:13.332626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:13.332641 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:13.332789 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:13.333401 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.334175 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.334923 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.335248 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.335310 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:13.337681 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:13.337690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:13.337690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:13.337697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:13.339421 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.339787 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.339790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.339790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.339859 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.340134 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.340134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.340134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.340274 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:13.340389 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:13.340468 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:59:13.340529 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:13.349025 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:13.342038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.2.165.126:36041)
0928 21:59:13.342144 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:13.342146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:13.342167 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:59:13.342188 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:13.342189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:13.342197 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:13.342238 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:13.342744 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.343228 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.344050 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:13.344267 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:13.344336 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:13.346700 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:13.346702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:13.346702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:13.346710 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:13.347961 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.348262 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.348263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.348263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.348311 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:13.348467 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:13.348468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:13.348468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:13.348615 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:13.348865 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:13.348898 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 21:59:13.348959 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 21:59:13.349314 6275 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44868
I20260928 21:59:13.349658 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6337
I20260928 21:59:13.354157 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6472
I20260928 21:59:13.358439 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6606
I20260928 21:59:13.362746 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6243
2026-09-28T21:59:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5737 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6747](info): set up 2 sockets
Sep 28 21:59:13 dist-test-slave-3t36 krb5kdc[6747](info): commencing operation
krb5kdc: starting...
W20260928 21:59:14.408727 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:14 dist-test-slave-3t36 krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:14.536111 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38865
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:14.620012 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:14.620265 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:14.620288 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:14.621469 6760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:14.621513 6760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:14.621527 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:14.621541 6760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:14.621551 6760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:14.623019 6760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:38865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:14.623243 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:14.623425 6760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:14.628647 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:14.628659 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:14.628768 6760 server_base.cc:1034] running on GCE node
W20260928 21:59:14.628659 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:14.629106 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:14.629374 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:14.630543 6760 hybrid_clock.cc:648] HybridClock initialized: now 1790632754630520 us; error 39 us; skew 500 ppm
Sep 28 21:59:14 dist-test-slave-3t36 krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:14.633105 6760 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:14.633558 6760 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:14.633630 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:14.633783 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:14.634871 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "083cfe6f0cf743cfab49a0ca843e7a2c"
format_stamp: "Formatted at 2026-09-28 21:59:14 on dist-test-slave-3t36"
server_key: "41e6ee32b455efdc333ea757cf409164"
server_key_iv: "ca50c1a0b95e7f93271ef67f62688c7e"
server_key_version: "encryptionkey@0"
I20260928 21:59:14.635245 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "083cfe6f0cf743cfab49a0ca843e7a2c"
format_stamp: "Formatted at 2026-09-28 21:59:14 on dist-test-slave-3t36"
server_key: "41e6ee32b455efdc333ea757cf409164"
server_key_iv: "ca50c1a0b95e7f93271ef67f62688c7e"
server_key_version: "encryptionkey@0"
I20260928 21:59:14.636554 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 21:59:14.637370 6774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:14.637545 6760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:14.637645 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "083cfe6f0cf743cfab49a0ca843e7a2c"
format_stamp: "Formatted at 2026-09-28 21:59:14 on dist-test-slave-3t36"
server_key: "41e6ee32b455efdc333ea757cf409164"
server_key_iv: "ca50c1a0b95e7f93271ef67f62688c7e"
server_key_version: "encryptionkey@0"
I20260928 21:59:14.637729 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:14.672899 6760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:14.675240 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:14.675422 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:14.679600 6760 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:38865
I20260928 21:59:14.679626 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:38865 every 8 connection(s)
I20260928 21:59:14.679932 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:14.681006 6827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:14.681532 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6760
I20260928 21:59:14.681643 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:14.681829 2709 external_mini_cluster.cc:1428] Setting key 6bccc4189e7fc5f619148d7de56abb4e
I20260928 21:59:14.684152 6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c: Bootstrap starting.
I20260928 21:59:14.684830 6827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:14.685168 6827 log.cc:824] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:14.685922 6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c: No bootstrap required, opened a new log
I20260928 21:59:14.688359 6827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38865 } }
I20260928 21:59:14.688496 6827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:14.688519 6827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083cfe6f0cf743cfab49a0ca843e7a2c, State: Initialized, Role: FOLLOWER
I20260928 21:59:14.688653 6827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38865 } }
I20260928 21:59:14.688771 6827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:14.688794 6827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:14.688863 6827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:59:14 dist-test-slave-3t36 krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:14.689440 6827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38865 } }
I20260928 21:59:14.689527 6827 leader_election.cc:302] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 083cfe6f0cf743cfab49a0ca843e7a2c; no voters:
I20260928 21:59:14.689752 6827 leader_election.cc:288] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:14.689921 6833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:14.690033 6827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:14.690109 6833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [term 1 LEADER]: Becoming Leader. State: Replica: 083cfe6f0cf743cfab49a0ca843e7a2c, State: Running, Role: LEADER
I20260928 21:59:14.690269 6833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38865 } }
I20260928 21:59:14.690675 6834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 083cfe6f0cf743cfab49a0ca843e7a2c. Latest consensus state: current_term: 1 leader_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38865 } } }
I20260928 21:59:14.690838 6834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:14.691056 6832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083cfe6f0cf743cfab49a0ca843e7a2c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38865 } } }
I20260928 21:59:14.691269 6832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:14.691478 6841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:14.692085 6841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:14.693256 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:14.682393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.2.165.126:38865)
0928 21:59:14.682552 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:14.682560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:14.682586 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:59:14.682995 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:14.683000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:14.683056 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:14.683174 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:14.683955 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:14.684971 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:14.686014 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:14.686460 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:14.686550 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:14.689439 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:14.689443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:14.689444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:14.689457 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:14.691609 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:14.691978 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:14.691981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:14.691982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:14.692080 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:14.692317 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:14.692318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:14.692318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:14.692564 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:14.692754 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:14.692871 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:59:14.693015 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260928 21:59:14.694036 6841 catalog_manager.cc:1347] Generated new cluster ID: 0f57784663554bd08bebf9cededf2503
I20260928 21:59:14.694101 6841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:14.700428 6841 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:14.701021 6841 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:14.707756 6841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c: Generated new TSK 0
I20260928 21:59:14.707964 6841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:59:14.743479 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38865
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:14.841262 6853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:14.841475 6853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:14.841500 6853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:14.842826 6853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:14.842871 6853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:14.842900 6853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:14.844460 6853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:14.844731 6853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:14.844991 6853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:14.850028 6861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:14.850040 6858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:14.850176 6859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:14.850715 6853 server_base.cc:1034] running on GCE node
I20260928 21:59:14.850930 6853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:14.851153 6853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:14.852361 6853 hybrid_clock.cc:648] HybridClock initialized: now 1790632754852367 us; error 48 us; skew 500 ppm
Sep 28 21:59:14 dist-test-slave-3t36 krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:14.855062 6853 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:59:14.855499 6853 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:14.855552 6853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:14.855648 6853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:14.856803 6853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190"
format_stamp: "Formatted at 2026-09-28 21:59:14 on dist-test-slave-3t36"
server_key: "d6251e53c2fe503aa52ddab47f6fef08"
server_key_iv: "29f22d81381298957d6c355b6e78ba44"
server_key_version: "encryptionkey@0"
I20260928 21:59:14.857116 6853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190"
format_stamp: "Formatted at 2026-09-28 21:59:14 on dist-test-slave-3t36"
server_key: "d6251e53c2fe503aa52ddab47f6fef08"
server_key_iv: "29f22d81381298957d6c355b6e78ba44"
server_key_version: "encryptionkey@0"
I20260928 21:59:14.858476 6853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:14.859211 6867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:14.859448 6853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:14.859524 6853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190"
format_stamp: "Formatted at 2026-09-28 21:59:14 on dist-test-slave-3t36"
server_key: "d6251e53c2fe503aa52ddab47f6fef08"
server_key_iv: "29f22d81381298957d6c355b6e78ba44"
server_key_version: "encryptionkey@0"
I20260928 21:59:14.859581 6853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:14.873310 6853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:14.875830 6853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:14.875952 6853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:14.876158 6853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:14.876420 6853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:14.876448 6853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:14.876472 6853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:14.876484 6853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:14.882670 6853 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:42949
I20260928 21:59:14.882712 6980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:42949 every 8 connection(s)
I20260928 21:59:14.882992 6853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:59:14 dist-test-slave-3t36 krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632754, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:14.889823 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6853
I20260928 21:59:14.889916 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:14.890142 2709 external_mini_cluster.cc:1428] Setting key fc0f3479e8d47a108f07f09e5545c522
I20260928 21:59:14.893649 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:14.883780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:58275 (local address 127.2.165.126:38865)
0928 21:59:14.883865 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:14.883867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:14.884013 (+ 146us) server_negotiation.cc:407] Connection header received
0928 21:59:14.884536 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:14.884538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:14.884548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:14.884606 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:14.885601 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:14.886112 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:14.887184 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:14.887445 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:14.887509 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:14.890076 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:14.890078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:14.890079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:14.890098 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:14.892049 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:14.892566 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:14.892568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:14.892568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:14.892637 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:14.892952 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:14.892953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:14.892954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:14.893079 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:14.893157 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:14.893462 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 21:59:14.893554 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:59:14.893918 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:14.883729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:38865 (local address 127.2.165.65:58275)
0928 21:59:14.883884 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:14.883922 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:14.884438 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:14.884696 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:14.884699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:14.884971 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:14.885482 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:14.885487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:14.886239 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:14.886240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:14.887081 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:14.887084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:14.887551 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:14.887552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:14.887678 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:14.888277 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:14.888286 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:14.889897 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:14.892145 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:14.892154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:14.892156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:14.892464 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:14.892742 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:14.892744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:14.892744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:14.892873 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:14.893147 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:14.893154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:14.893315 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 21:59:14.893427 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:59:14.893606 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260928 21:59:14.894428 6981 heartbeater.cc:347] Connected to a master server at 127.2.165.126:38865
I20260928 21:59:14.894526 6981 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:14.894703 6981 heartbeater.cc:511] Master 127.2.165.126:38865 requested a full tablet report, sending...
I20260928 21:59:14.895231 6791 ts_manager.cc:194] Registered new tserver with Master: ce8dfdebd2b94a6aa4ecdacdd33f2190 (127.2.165.65:42949)
I20260928 21:59:14.895879 6791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:58275
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:59:14.920264 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38865
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:15.005002 6986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:15.005254 6986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:15.005282 6986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:15.006532 6986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:15.006578 6986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:15.006608 6986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:15.008142 6986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:15.008354 6986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:15.008549 6986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:15.013945 6992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.013952 6991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.013945 6994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:15.013968 6986 server_base.cc:1034] running on GCE node
I20260928 21:59:15.014492 6986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:15.014757 6986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:15.015919 6986 hybrid_clock.cc:648] HybridClock initialized: now 1790632755015887 us; error 47 us; skew 500 ppm
Sep 28 21:59:15 dist-test-slave-3t36 krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:15.026481 6986 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:59:15.026958 6986 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:15.027048 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:15.027226 6986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:15.028429 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "75ecd618d67808aed54626bee4990666"
server_key_iv: "206df3476e4ad1cea5280aca384a8acb"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.028860 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "75ecd618d67808aed54626bee4990666"
server_key_iv: "206df3476e4ad1cea5280aca384a8acb"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.030329 6986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:15.031144 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.031359 6986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:15.031481 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "75ecd618d67808aed54626bee4990666"
server_key_iv: "206df3476e4ad1cea5280aca384a8acb"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.031559 6986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:15.066679 6986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:15.069015 6986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:15.069248 6986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:15.069517 6986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:15.069872 6986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:15.069931 6986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.069975 6986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:15.070008 6986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.075959 6986 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:35203
I20260928 21:59:15.075999 7113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:35203 every 8 connection(s)
I20260928 21:59:15.076313 6986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:59:15 dist-test-slave-3t36 krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:15.086190 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 6986
I20260928 21:59:15.086311 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:15.086516 2709 external_mini_cluster.cc:1428] Setting key 5fc6fc32fc522284ff6c0c94ceb32c4c
I20260928 21:59:15.086675 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.077134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:38933 (local address 127.2.165.126:38865)
0928 21:59:15.077215 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.077218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.077469 (+ 251us) server_negotiation.cc:407] Connection header received
0928 21:59:15.077958 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.077959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.077969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.078029 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:15.079050 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.079603 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.080637 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.080934 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.081000 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.083376 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:15.083377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:15.083378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:15.083385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:15.084945 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:15.085435 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:15.085436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:15.085437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:15.085514 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:15.085780 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:15.085782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:15.085782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:15.085904 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:15.085970 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.086465 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.086547 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:59:15.086941 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.077123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:38865 (local address 127.2.165.66:38933)
0928 21:59:15.077336 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.077375 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.077863 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.078110 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.078114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.078408 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:15.078939 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.078945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.079697 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.079698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.080530 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.080532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.081021 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.081022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.081148 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.081718 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:15.081727 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:15.083246 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:15.085035 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:15.085042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:15.085044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:15.085325 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:15.085581 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:15.085582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:15.085582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:15.085697 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:15.086025 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:15.086031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:15.086272 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.086435 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.086639 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":124,"threads_started":1}
I20260928 21:59:15.087464 7114 heartbeater.cc:347] Connected to a master server at 127.2.165.126:38865
I20260928 21:59:15.087579 7114 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:15.087738 7114 heartbeater.cc:511] Master 127.2.165.126:38865 requested a full tablet report, sending...
I20260928 21:59:15.088195 6791 ts_manager.cc:194] Registered new tserver with Master: 5a9bce6c8fad4f3ea8cde4f56ee58928 (127.2.165.66:35203)
I20260928 21:59:15.088582 6791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:38933
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:59:15.120239 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38865
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:15.219137 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:15.219399 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:15.219442 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:15.220651 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:15.220768 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:15.220816 7119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:15.222373 7119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:15.222649 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:15.222891 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:15.228475 7124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.228466 7125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:15.228499 7119 server_base.cc:1034] running on GCE node
W20260928 21:59:15.228466 7127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:15.228968 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:15.229306 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:15.230487 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790632755230479 us; error 37 us; skew 500 ppm
Sep 28 21:59:15 dist-test-slave-3t36 krb5kdc[6747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:15.233330 7119 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:59:15.233714 7119 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:15.233788 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:15.233963 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:15.235159 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "de4e997cdd0a4a0598fd2ba599ac6fab"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "64ee1d59acd87f72fcd362c5448bdb25"
server_key_iv: "6244767734bf29391ad25e7a735033d7"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.235556 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "de4e997cdd0a4a0598fd2ba599ac6fab"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "64ee1d59acd87f72fcd362c5448bdb25"
server_key_iv: "6244767734bf29391ad25e7a735033d7"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.237032 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:59:15.237975 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.238242 7119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:15.238359 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "de4e997cdd0a4a0598fd2ba599ac6fab"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "64ee1d59acd87f72fcd362c5448bdb25"
server_key_iv: "6244767734bf29391ad25e7a735033d7"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.238451 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:15.263422 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:15.265945 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:15.266124 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:15.266441 7119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:15.266763 7119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:15.266817 7119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.266882 7119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:15.266927 7119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.272914 7119 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:35645
I20260928 21:59:15.272965 7246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:35645 every 8 connection(s)
I20260928 21:59:15.273285 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:15.276728 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 7119
I20260928 21:59:15.276829 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:15.277019 2709 external_mini_cluster.cc:1428] Setting key 4ec4377386f25558d6f948ef6ea1f10f
Sep 28 21:59:15 dist-test-slave-3t36 krb5kdc[6747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632755, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:15.283730 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.274089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:53561 (local address 127.2.165.126:38865)
0928 21:59:15.274149 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.274151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.274411 (+ 260us) server_negotiation.cc:407] Connection header received
0928 21:59:15.275007 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.275009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.275022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.275073 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:15.276060 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.276573 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.278086 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.278296 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.278386 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.280734 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:15.280736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:15.280736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:15.280744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:15.282265 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:15.282709 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:15.282710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:15.282711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:15.282775 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:15.283043 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:15.283044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:15.283044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:15.283167 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:15.283243 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.283534 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.283618 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:59:15.283929 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.274058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:38865 (local address 127.2.165.67:53561)
0928 21:59:15.274283 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.274321 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.274884 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.275153 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.275157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.275438 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:15.275954 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.275959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.276707 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.276708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.277981 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.277984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.278395 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.278397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.278541 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.279128 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:15.279137 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:15.280589 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:15.282377 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:15.282384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:15.282387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:15.282627 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:15.282844 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:15.282845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:15.282845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:15.282968 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:15.283242 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:15.283249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:15.283397 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.283512 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.283682 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260928 21:59:15.284714 7247 heartbeater.cc:347] Connected to a master server at 127.2.165.126:38865
I20260928 21:59:15.284816 7247 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:15.284950 7247 heartbeater.cc:511] Master 127.2.165.126:38865 requested a full tablet report, sending...
I20260928 21:59:15.285368 6791 ts_manager.cc:194] Registered new tserver with Master: de4e997cdd0a4a0598fd2ba599ac6fab (127.2.165.67:35645)
I20260928 21:59:15.285781 6791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:53561
I20260928 21:59:15.288820 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:59:15.297108 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.290222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.2.165.126:38865)
0928 21:59:15.290287 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.290289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.290444 (+ 155us) server_negotiation.cc:407] Connection header received
0928 21:59:15.290482 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.290483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.290512 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.290575 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:15.291251 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.291754 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.292572 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.292844 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.292906 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.294137 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:15.294138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:15.294139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:15.294145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:15.295650 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:15.296052 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:15.296053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:15.296054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:15.296113 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:15.296409 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:15.296410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:15.296410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:15.296589 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:15.296745 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.296916 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.296989 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:59:15.299186 6791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:15.300997 6791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:15.308310 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.304114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.2.165.66:35203)
0928 21:59:15.304378 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.304383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.304394 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:15.304435 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.304437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.304453 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.304530 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:15.304831 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.305516 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.307042 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.307831 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.307909 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.307988 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.308153 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.308204 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260928 21:59:15.308310 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.303564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:35645 (local address 127.0.0.1:40020)
0928 21:59:15.303710 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.303760 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.303839 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.304591 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.304592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.304594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:15.304628 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.304629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.306061 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.306062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.307236 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.307239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.307980 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.307980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.308033 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.308046 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.308099 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.308189 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":46,"thread_start_us":62,"threads_started":1}
I20260928 21:59:15.308314 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.303992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:35203 (local address 127.0.0.1:38092)
0928 21:59:15.304273 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.304281 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.304317 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.304619 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.304620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.304622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:15.304737 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.304740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.305631 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.305632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.306934 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.306937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.307924 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.307925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.308028 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.308046 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.308125 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.308190 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260928 21:59:15.308460 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.303800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:42949 (local address 127.0.0.1:47582)
0928 21:59:15.304119 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.304130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.304182 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.304496 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.304500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.304502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:15.304605 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.304607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.306092 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.306093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.307270 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.307273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.308232 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.308233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.308296 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.308300 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.308332 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.308378 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260928 21:59:15.308725 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.303727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.2.165.67:35645)
0928 21:59:15.304434 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.304439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.304448 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:15.304486 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.304488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.304501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.304719 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:15.304806 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.305947 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.307324 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.307861 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.308450 (+ 589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.308522 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.308551 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.308614 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":109,"threads_started":1}
I20260928 21:59:15.308835 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.303941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.2.165.65:42949)
0928 21:59:15.304159 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.304164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.304177 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:59:15.304350 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.304353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.304373 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.304717 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:15.304804 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.305948 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.307379 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.308134 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.308560 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.308639 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.308668 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.308732 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
I20260928 21:59:15.309465 7181 tablet_service.cc:1467] Processing CreateTablet for tablet f2dc8e4c051b40c4aa40355a71533848 (DEFAULT_TABLE table=test-table [id=bcf0aa482c174b728f049907751c7c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:15.309464 6915 tablet_service.cc:1467] Processing CreateTablet for tablet f2dc8e4c051b40c4aa40355a71533848 (DEFAULT_TABLE table=test-table [id=bcf0aa482c174b728f049907751c7c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:15.309696 7048 tablet_service.cc:1467] Processing CreateTablet for tablet f2dc8e4c051b40c4aa40355a71533848 (DEFAULT_TABLE table=test-table [id=bcf0aa482c174b728f049907751c7c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:15.309844 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dc8e4c051b40c4aa40355a71533848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:15.309844 7181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dc8e4c051b40c4aa40355a71533848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:15.310045 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2dc8e4c051b40c4aa40355a71533848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:15.313021 7264 tablet_bootstrap.cc:492] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: Bootstrap starting.
I20260928 21:59:15.313342 7266 tablet_bootstrap.cc:492] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190: Bootstrap starting.
I20260928 21:59:15.313342 7265 tablet_bootstrap.cc:492] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Bootstrap starting.
I20260928 21:59:15.313735 7264 tablet_bootstrap.cc:654] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:15.313997 7266 tablet_bootstrap.cc:654] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:15.313997 7265 tablet_bootstrap.cc:654] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:15.314325 7264 log.cc:824] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:15.314637 7265 log.cc:824] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:15.314637 7266 log.cc:824] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:15.315136 7264 tablet_bootstrap.cc:492] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: No bootstrap required, opened a new log
I20260928 21:59:15.315217 7264 ts_tablet_manager.cc:1397] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:15.315529 7265 tablet_bootstrap.cc:492] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: No bootstrap required, opened a new log
I20260928 21:59:15.315625 7265 ts_tablet_manager.cc:1397] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:15.315716 7266 tablet_bootstrap.cc:492] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190: No bootstrap required, opened a new log
I20260928 21:59:15.315796 7266 ts_tablet_manager.cc:1397] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:15.317130 7265 raft_consensus.cc:348] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.317303 7265 raft_consensus.cc:374] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:15.317345 7265 raft_consensus.cc:729] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4e997cdd0a4a0598fd2ba599ac6fab, State: Initialized, Role: FOLLOWER
I20260928 21:59:15.317498 7265 consensus_queue.cc:260] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.317788 7265 ts_tablet_manager.cc:1428] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:59:15.317925 7247 heartbeater.cc:503] Master 127.2.165.126:38865 was elected leader, sending a full tablet report...
I20260928 21:59:15.317866 7266 raft_consensus.cc:348] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.318037 7266 raft_consensus.cc:374] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:15.318109 7266 raft_consensus.cc:729] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8dfdebd2b94a6aa4ecdacdd33f2190, State: Initialized, Role: FOLLOWER
I20260928 21:59:15.318262 7266 consensus_queue.cc:260] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.318590 6981 heartbeater.cc:503] Master 127.2.165.126:38865 was elected leader, sending a full tablet report...
I20260928 21:59:15.317507 7264 raft_consensus.cc:348] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.318800 7264 raft_consensus.cc:374] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:15.318861 7264 raft_consensus.cc:729] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9bce6c8fad4f3ea8cde4f56ee58928, State: Initialized, Role: FOLLOWER
I20260928 21:59:15.318594 7266 ts_tablet_manager.cc:1428] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:59:15.319008 7264 consensus_queue.cc:260] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.319306 7264 ts_tablet_manager.cc:1428] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 21:59:15.319454 7114 heartbeater.cc:503] Master 127.2.165.126:38865 was elected leader, sending a full tablet report...
W20260928 21:59:15.326963 7115 tablet.cc:2369] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:59:15.383677 6982 tablet.cc:2369] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:15.457320 7270 raft_consensus.cc:482] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:15.457453 7270 raft_consensus.cc:504] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.457772 7270 leader_election.cc:288] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce8dfdebd2b94a6aa4ecdacdd33f2190 (127.2.165.65:42949), 5a9bce6c8fad4f3ea8cde4f56ee58928 (127.2.165.66:35203)
I20260928 21:59:15.462194 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.457996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:42949 (local address 127.2.165.67:58621)
0928 21:59:15.458116 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.458132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.458202 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.458428 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.458430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.458432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:15.458525 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.458527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.459495 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.459496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.461000 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.461001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.461865 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.461866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.461961 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.461964 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.462022 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.462079 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":123}
I20260928 21:59:15.462253 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.458304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:52009 (local address 127.2.165.66:35203)
0928 21:59:15.458383 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.458385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.458395 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:15.458460 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.458460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.458474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.458518 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:15.458673 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.459381 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.461138 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.461750 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.461841 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.461851 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.462075 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.462169 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:59:15.462271 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.458153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:58621 (local address 127.2.165.65:42949)
0928 21:59:15.458255 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.458258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.458269 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:15.458300 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.458301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.458318 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.458409 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:15.458610 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.459381 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.461138 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.461761 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.461841 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.461851 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.462075 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.462128 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:59:15.462200 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.458135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:35203 (local address 127.2.165.67:52009)
0928 21:59:15.458318 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.458337 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.458402 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.458539 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.458540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.458542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:15.458600 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.458602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.459495 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.459496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.460997 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:15.461001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.461865 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.461866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:15.461950 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.461955 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.462022 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.462079 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":92,"thread_start_us":119,"threads_started":1}
I20260928 21:59:15.462749 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dc8e4c051b40c4aa40355a71533848" candidate_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" is_pre_election: true
I20260928 21:59:15.462790 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dc8e4c051b40c4aa40355a71533848" candidate_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" is_pre_election: true
I20260928 21:59:15.462942 6935 raft_consensus.cc:2454] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de4e997cdd0a4a0598fd2ba599ac6fab in term 0.
I20260928 21:59:15.462951 7068 raft_consensus.cc:2454] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de4e997cdd0a4a0598fd2ba599ac6fab in term 0.
I20260928 21:59:15.463192 7135 leader_election.cc:302] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce8dfdebd2b94a6aa4ecdacdd33f2190, de4e997cdd0a4a0598fd2ba599ac6fab; no voters:
I20260928 21:59:15.463376 7270 raft_consensus.cc:2781] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:15.463454 7270 raft_consensus.cc:482] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:15.463501 7270 raft_consensus.cc:3037] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:15.464241 7270 raft_consensus.cc:504] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.464386 7270 leader_election.cc:288] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [CANDIDATE]: Term 1 election: Requested vote from peers ce8dfdebd2b94a6aa4ecdacdd33f2190 (127.2.165.65:42949), 5a9bce6c8fad4f3ea8cde4f56ee58928 (127.2.165.66:35203)
I20260928 21:59:15.464566 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dc8e4c051b40c4aa40355a71533848" candidate_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190"
I20260928 21:59:15.464607 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2dc8e4c051b40c4aa40355a71533848" candidate_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928"
I20260928 21:59:15.464636 6935 raft_consensus.cc:3037] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:15.464727 7068 raft_consensus.cc:3037] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:15.465358 6935 raft_consensus.cc:2454] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de4e997cdd0a4a0598fd2ba599ac6fab in term 1.
I20260928 21:59:15.465523 7135 leader_election.cc:302] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce8dfdebd2b94a6aa4ecdacdd33f2190, de4e997cdd0a4a0598fd2ba599ac6fab; no voters:
I20260928 21:59:15.465555 7068 raft_consensus.cc:2454] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de4e997cdd0a4a0598fd2ba599ac6fab in term 1.
I20260928 21:59:15.465628 7270 raft_consensus.cc:2781] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:15.465809 7270 raft_consensus.cc:686] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 1 LEADER]: Becoming Leader. State: Replica: de4e997cdd0a4a0598fd2ba599ac6fab, State: Running, Role: LEADER
I20260928 21:59:15.465920 7270 consensus_queue.cc:237] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } }
I20260928 21:59:15.466812 6791 catalog_manager.cc:5495] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab reported cstate change: term changed from 0 to 1, leader changed from <none> to de4e997cdd0a4a0598fd2ba599ac6fab (127.2.165.67). New cstate: current_term: 1 leader_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4e997cdd0a4a0598fd2ba599ac6fab" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:15.491488 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.488726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.2.165.67:35645)
0928 21:59:15.488780 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.488782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.488792 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:15.488867 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.488868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.488886 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.488970 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:15.489120 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.489669 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.490574 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:15.490779 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:15.490858 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:15.491037 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:15.491139 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:15.491298 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.491317 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.491338 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:59:15.493539 7068 raft_consensus.cc:1261] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 1 FOLLOWER]: Refusing update from remote peer de4e997cdd0a4a0598fd2ba599ac6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:15.493539 6935 raft_consensus.cc:1261] T f2dc8e4c051b40c4aa40355a71533848 P ce8dfdebd2b94a6aa4ecdacdd33f2190 [term 1 FOLLOWER]: Refusing update from remote peer de4e997cdd0a4a0598fd2ba599ac6fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:15.493954 7270 consensus_queue.cc:1035] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce8dfdebd2b94a6aa4ecdacdd33f2190" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:15.494144 7274 consensus_queue.cc:1035] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9bce6c8fad4f3ea8cde4f56ee58928" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:15.497130 7279 mvcc.cc:205] Tried to move back new op lower bound from 7334431766498484224 to 7334431766389055488. Current Snapshot: MvccSnapshot[applied={T|T < 7334431766498484224}]
I20260928 21:59:15.498769 6789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:15.498878 6789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:15.499738 6789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 083cfe6f0cf743cfab49a0ca843e7a2c: Sending DeleteTablet for 3 replicas of tablet f2dc8e4c051b40c4aa40355a71533848
I20260928 21:59:15.500128 7181 tablet_service.cc:1514] Processing DeleteTablet for tablet f2dc8e4c051b40c4aa40355a71533848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:15 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:40020
I20260928 21:59:15.500203 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet f2dc8e4c051b40c4aa40355a71533848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:15 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:47582
I20260928 21:59:15.500284 7048 tablet_service.cc:1514] Processing DeleteTablet for tablet f2dc8e4c051b40c4aa40355a71533848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:15 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:38092
I20260928 21:59:15.500584 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6853
I20260928 21:59:15.504863 7286 tablet_replica.cc:330] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: stopping tablet replica
I20260928 21:59:15.505029 7286 raft_consensus.cc:2229] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:15.505183 7286 raft_consensus.cc:2258] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:15.505366 7288 tablet_replica.cc:330] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: stopping tablet replica
I20260928 21:59:15.505496 7288 raft_consensus.cc:2229] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:15.505600 7288 raft_consensus.cc:2258] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:15.506165 7286 ts_tablet_manager.cc:1905] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:15.506165 7288 ts_tablet_manager.cc:1905] T f2dc8e4c051b40c4aa40355a71533848 P 5a9bce6c8fad4f3ea8cde4f56ee58928: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:59:15.507364 6776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:42949 (error 108)
I20260928 21:59:15.507534 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6986
I20260928 21:59:15.509686 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.509503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:42949 (local address 127.0.0.1:47598)
0928 21:59:15.509582 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.509640 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 21:59:15.509836 6776 catalog_manager.cc:4584] TS ce8dfdebd2b94a6aa4ecdacdd33f2190 (127.2.165.65:42949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2dc8e4c051b40c4aa40355a71533848: Network error: Client connection negotiation failed: client connection to 127.2.165.65:42949: connect: Connection refused (error 111)
I20260928 21:59:15.510497 7286 ts_tablet_manager.cc:1918] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:15.510586 7286 log.cc:1196] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/f2dc8e4c051b40c4aa40355a71533848
I20260928 21:59:15.510885 7286 ts_tablet_manager.cc:1939] T f2dc8e4c051b40c4aa40355a71533848 P de4e997cdd0a4a0598fd2ba599ac6fab: Deleting consensus metadata
I20260928 21:59:15.511462 6776 catalog_manager.cc:4841] TS de4e997cdd0a4a0598fd2ba599ac6fab (127.2.165.67:35645): tablet f2dc8e4c051b40c4aa40355a71533848 (table test-table [id=bcf0aa482c174b728f049907751c7c30]) successfully deleted
I20260928 21:59:15.512840 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 7119
I20260928 21:59:15.512928 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.512817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:35203 (local address 127.0.0.1:38094)
0928 21:59:15.512861 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.512887 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.66:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 21:59:15.517764 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 6760
2026-09-28T21:59:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2156 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T21:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:15Z Disabled control of system clock
I20260928 21:59:15.546881 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:39381
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:59:15.632321 7293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:15.632535 7293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:15.632558 7293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:15.633760 7293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:15.633807 7293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:15.633821 7293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:15.633833 7293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:15.633845 7293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:15.635226 7293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:39381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:15.635473 7293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:15.635788 7293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:15.640803 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:15.640856 7293 server_base.cc:1034] running on GCE node
W20260928 21:59:15.640851 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.640838 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:15.641333 7293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:15.641645 7293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:15.642818 7293 hybrid_clock.cc:648] HybridClock initialized: now 1790632755642800 us; error 51 us; skew 500 ppm
I20260928 21:59:15.644243 7293 webserver.cc:460] Webserver started at http://127.2.165.126:35601/ using document root <none> and password file <none>
I20260928 21:59:15.644515 7293 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:15.644588 7293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:15.644794 7293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:15.645885 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "9369362222bb4dd080304b9f648c3f10"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "8690f3a55002f7aa98d921c971574b29"
server_key_iv: "10e512ed3bb1159ffe5b3b24b09fc899"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.646281 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "9369362222bb4dd080304b9f648c3f10"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "8690f3a55002f7aa98d921c971574b29"
server_key_iv: "10e512ed3bb1159ffe5b3b24b09fc899"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.647599 7293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:15.648331 7307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.648515 7293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:15.648607 7293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "9369362222bb4dd080304b9f648c3f10"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "8690f3a55002f7aa98d921c971574b29"
server_key_iv: "10e512ed3bb1159ffe5b3b24b09fc899"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.648737 7293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:15.652151 7293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:15.652442 7293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:15.652590 7293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:15.656490 7293 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:39381
I20260928 21:59:15.656540 7359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:39381 every 8 connection(s)
I20260928 21:59:15.656932 7293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:15.657629 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:15.660399 7360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10: Bootstrap starting.
I20260928 21:59:15.661016 7360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:15.661252 7360 log.cc:824] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:15.661872 7360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10: No bootstrap required, opened a new log
I20260928 21:59:15.662251 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 7293
I20260928 21:59:15.662357 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:15.662547 2709 external_mini_cluster.cc:1428] Setting key acbad98f7a28dd80b2f30be35b7d6103
I20260928 21:59:15.663998 7360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9369362222bb4dd080304b9f648c3f10" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39381 } }
I20260928 21:59:15.664189 7360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:15.664232 7360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9369362222bb4dd080304b9f648c3f10, State: Initialized, Role: FOLLOWER
I20260928 21:59:15.664410 7360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9369362222bb4dd080304b9f648c3f10" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39381 } }
I20260928 21:59:15.664497 7360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:15.664520 7360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:15.664577 7360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:15.665652 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.663120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.2.165.126:39381)
0928 21:59:15.663327 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.663335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.663361 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:59:15.663930 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.663937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.663979 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.664098 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:15.665066 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:15.665075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:15.665075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:15.665079 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:15.665131 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:15.665147 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:15.665220 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.665336 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.665385 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260928 21:59:15.665901 7360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9369362222bb4dd080304b9f648c3f10" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39381 } }
I20260928 21:59:15.666015 7360 leader_election.cc:302] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9369362222bb4dd080304b9f648c3f10; no voters:
I20260928 21:59:15.666211 7360 leader_election.cc:288] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:15.666246 7365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:15.666407 7365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [term 1 LEADER]: Becoming Leader. State: Replica: 9369362222bb4dd080304b9f648c3f10, State: Running, Role: LEADER
I20260928 21:59:15.666473 7360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:15.666505 7365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9369362222bb4dd080304b9f648c3f10" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39381 } }
I20260928 21:59:15.666882 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9369362222bb4dd080304b9f648c3f10. Latest consensus state: current_term: 1 leader_uuid: "9369362222bb4dd080304b9f648c3f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9369362222bb4dd080304b9f648c3f10" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39381 } } }
I20260928 21:59:15.666976 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:15.666898 7367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9369362222bb4dd080304b9f648c3f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9369362222bb4dd080304b9f648c3f10" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39381 } } }
I20260928 21:59:15.667151 7367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10 [sys.catalog]: This master's current role is: LEADER
W20260928 21:59:15.668257 7381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:59:15.668320 7381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:59:15.668421 7372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:15.668604 7372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:15.670064 7372 catalog_manager.cc:1347] Generated new cluster ID: 0e51297e4ab04b1786ee3edb09ad6642
I20260928 21:59:15.670117 7372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:15.690037 7372 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:15.690258 7372 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:15.695676 7372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10: Generated new TSK 0
I20260928 21:59:15.695902 7372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:59:15.702932 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39381
--builtin_ntp_servers=127.2.165.84:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:59:15.797340 7384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:15.797560 7384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:15.797585 7384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:15.798925 7384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:15.798975 7384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:15.799002 7384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:15.800400 7384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:15.800662 7384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:15.801007 7384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:15.807255 7389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.807287 7390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.807525 7392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:15.807559 7384 server_base.cc:1034] running on GCE node
I20260928 21:59:15.807914 7384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:15.808177 7384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:15.809372 7384 hybrid_clock.cc:648] HybridClock initialized: now 1790632755809351 us; error 34 us; skew 500 ppm
I20260928 21:59:15.810726 7384 webserver.cc:460] Webserver started at http://127.2.165.65:41891/ using document root <none> and password file <none>
I20260928 21:59:15.811003 7384 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:15.811079 7384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:15.811231 7384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:15.812356 7384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "37a70a24b35b4647b5a738de820b5efb"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "4f0e7a2012c7e9367379ea4eac4fdb3a"
server_key_iv: "9a16b1355fa5fd681a861dca3d4863b4"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.812788 7384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "37a70a24b35b4647b5a738de820b5efb"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "4f0e7a2012c7e9367379ea4eac4fdb3a"
server_key_iv: "9a16b1355fa5fd681a861dca3d4863b4"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.814208 7384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:15.815045 7398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.815264 7384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:15.815363 7384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "37a70a24b35b4647b5a738de820b5efb"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "4f0e7a2012c7e9367379ea4eac4fdb3a"
server_key_iv: "9a16b1355fa5fd681a861dca3d4863b4"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.815446 7384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:15.818997 7384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:15.819267 7384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:15.819412 7384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:15.819622 7384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:15.819923 7384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:15.819976 7384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.820014 7384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:15.820041 7384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.825732 7384 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:35673
I20260928 21:59:15.825811 7511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:35673 every 8 connection(s)
I20260928 21:59:15.826152 7384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:15.828755 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 7384
I20260928 21:59:15.828871 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:15.829099 2709 external_mini_cluster.cc:1428] Setting key 6524500a38edc31c5953c0648665f110
I20260928 21:59:15.829654 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.826917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:37781 (local address 127.2.165.126:39381)
0928 21:59:15.826982 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.826985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.827191 (+ 206us) server_negotiation.cc:407] Connection header received
0928 21:59:15.827713 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.827714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.827722 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.827780 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:15.829414 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:15.829416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:15.829416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:15.829421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:15.829437 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:15.829441 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:15.829493 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.829569 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.829585 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:59:15.829907 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.826878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39381 (local address 127.2.165.65:37781)
0928 21:59:15.827064 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.827098 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.827603 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.827885 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.827893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.827902 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:15.829221 (+ 1319us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:15.829221 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:15.829263 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:15.829302 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:15.829490 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:15.829496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:15.829499 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.829580 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.829645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260928 21:59:15.830238 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39381
--builtin_ntp_servers=127.2.165.84:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 21:59:15.831540 7512 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39381
I20260928 21:59:15.831658 7512 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:15.831813 7512 heartbeater.cc:511] Master 127.2.165.126:39381 requested a full tablet report, sending...
I20260928 21:59:15.832365 7324 ts_manager.cc:194] Registered new tserver with Master: 37a70a24b35b4647b5a738de820b5efb (127.2.165.65:35673)
W20260928 21:59:15.914045 7515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:15.914318 7515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:15.914361 7515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:15.915637 7515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:15.915712 7515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:15.915755 7515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:15.917300 7515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:15.917591 7515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:15.917905 7515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:15.923384 7523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.923316 7520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:15.923316 7521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:15.923461 7515 server_base.cc:1034] running on GCE node
I20260928 21:59:15.923822 7515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:15.924057 7515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:15.925197 7515 hybrid_clock.cc:648] HybridClock initialized: now 1790632755925174 us; error 43 us; skew 500 ppm
I20260928 21:59:15.926671 7515 webserver.cc:460] Webserver started at http://127.2.165.66:43867/ using document root <none> and password file <none>
I20260928 21:59:15.926927 7515 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:15.926996 7515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:15.927155 7515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:15.928246 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "218a5b50734e4ef3a9d4540fd0462096"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "e0c1e437506dba8f21f0b652a06b12eb"
server_key_iv: "3cff1dbca9555799f96c56b007138194"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.928623 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "218a5b50734e4ef3a9d4540fd0462096"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "e0c1e437506dba8f21f0b652a06b12eb"
server_key_iv: "3cff1dbca9555799f96c56b007138194"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.930023 7515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:15.930855 7529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.931061 7515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:15.931162 7515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "218a5b50734e4ef3a9d4540fd0462096"
format_stamp: "Formatted at 2026-09-28 21:59:15 on dist-test-slave-3t36"
server_key: "e0c1e437506dba8f21f0b652a06b12eb"
server_key_iv: "3cff1dbca9555799f96c56b007138194"
server_key_version: "encryptionkey@0"
I20260928 21:59:15.931246 7515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:15.934799 7515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:15.935096 7515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:15.935235 7515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:15.935446 7515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:15.935753 7515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:15.935806 7515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.935838 7515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:15.935865 7515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:15.941751 7515 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:37583
I20260928 21:59:15.941792 7642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:37583 every 8 connection(s)
I20260928 21:59:15.942126 7515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:15.945112 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.942856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:58131 (local address 127.2.165.126:39381)
0928 21:59:15.942932 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:15.942935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:15.943163 (+ 228us) server_negotiation.cc:407] Connection header received
0928 21:59:15.943607 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:15.943608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:15.943614 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:15.943664 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:15.944869 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:15.944871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:15.944871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:15.944876 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:15.944890 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:15.944894 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:15.944947 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:15.945002 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:59:15.945015 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:59:15.945310 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:15.942833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39381 (local address 127.2.165.66:58131)
0928 21:59:15.943040 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:15.943075 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:15.943527 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:15.943726 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:15.943729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:15.943738 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:15.944638 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:15.944638 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:15.944714 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:15.944753 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:15.944952 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:15.944955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:15.944957 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:59:15.945010 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:59:15.945083 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260928 21:59:15.945318 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 7515
I20260928 21:59:15.945431 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:15.945639 2709 external_mini_cluster.cc:1428] Setting key caebce1d7a4790a50bda9c788a4138c1
I20260928 21:59:15.945789 7643 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39381
I20260928 21:59:15.945905 7643 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:15.946064 7643 heartbeater.cc:511] Master 127.2.165.126:39381 requested a full tablet report, sending...
I20260928 21:59:15.946458 7324 ts_manager.cc:194] Registered new tserver with Master: 218a5b50734e4ef3a9d4540fd0462096 (127.2.165.66:37583)
I20260928 21:59:15.946707 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39381
--builtin_ntp_servers=127.2.165.84:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:59:16.029139 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:16.029357 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:16.029381 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:16.030504 7646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:16.030541 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:16.030568 7646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:16.031957 7646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:16.032172 7646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:16.032428 7646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:16.037678 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:16.037669 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:16.037703 7646 server_base.cc:1034] running on GCE node
W20260928 21:59:16.037669 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:16.038125 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:16.038393 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:16.039559 7646 hybrid_clock.cc:648] HybridClock initialized: now 1790632756039543 us; error 36 us; skew 500 ppm
I20260928 21:59:16.041101 7646 webserver.cc:460] Webserver started at http://127.2.165.67:41703/ using document root <none> and password file <none>
I20260928 21:59:16.041383 7646 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:16.041435 7646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:16.041534 7646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:16.042621 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "0b7210e58da84687ad955bbb409db4a2"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "64648eecb0476241009d4ddee7beace8"
server_key_iv: "f1a0ec28c35dbe11fd61a20bbf51008c"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.043057 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "0b7210e58da84687ad955bbb409db4a2"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "64648eecb0476241009d4ddee7beace8"
server_key_iv: "f1a0ec28c35dbe11fd61a20bbf51008c"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.044569 7646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:16.045528 7660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:16.045702 7646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:16.045833 7646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "0b7210e58da84687ad955bbb409db4a2"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "64648eecb0476241009d4ddee7beace8"
server_key_iv: "f1a0ec28c35dbe11fd61a20bbf51008c"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.045933 7646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:16.049454 7646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:16.049736 7646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:16.049887 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:16.050096 7646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:16.050427 7646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:16.050483 7646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:16.050523 7646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:16.050554 7646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:16.056669 7646 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:43605
I20260928 21:59:16.056722 7773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:43605 every 8 connection(s)
I20260928 21:59:16.057109 7646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:16.060268 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.057879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:33815 (local address 127.2.165.126:39381)
0928 21:59:16.057957 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.057959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.058173 (+ 214us) server_negotiation.cc:407] Connection header received
0928 21:59:16.058641 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.058642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.058647 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.058700 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.059961 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.059963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.059963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.059968 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.059984 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.059988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.060047 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.060184 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.060198 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:59:16.060492 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.057848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39381 (local address 127.2.165.67:33815)
0928 21:59:16.058053 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.058090 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.058559 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.058753 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.058756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.058765 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:16.059769 (+ 1004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:16.059769 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:16.059801 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:16.059854 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:16.060075 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:16.060083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:16.060086 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:16.060190 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:59:16.060247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260928 21:59:16.061030 7774 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39381
I20260928 21:59:16.061138 7774 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:16.061283 7774 heartbeater.cc:511] Master 127.2.165.126:39381 requested a full tablet report, sending...
I20260928 21:59:16.061713 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 7646
I20260928 21:59:16.061723 7324 ts_manager.cc:194] Registered new tserver with Master: 0b7210e58da84687ad955bbb409db4a2 (127.2.165.67:43605)
I20260928 21:59:16.061811 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:16.062031 2709 external_mini_cluster.cc:1428] Setting key 4e4ea4c69a6d486b2ab767f4cd9486c2
I20260928 21:59:16.062754 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:59:16.065346 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.064107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.2.165.126:39381)
0928 21:59:16.064188 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.064190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.064263 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:59:16.064342 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.064343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.064349 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.064398 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.065053 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.065054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.065054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.065058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.065072 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.065076 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.065121 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.065254 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.065270 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:59:16.067392 7324 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:16.069247 7324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:16.073606 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.071609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:37583 (local address 127.0.0.1:49970)
0928 21:59:16.072019 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.072045 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.072126 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.072308 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.072314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.072321 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:16.073145 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:16.073146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:16.073163 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:16.073191 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:16.073450 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:16.073456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:16.073459 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:16.073527 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:59:16.073540 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
I20260928 21:59:16.073904 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.071708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.2.165.66:37583)
0928 21:59:16.071924 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.071930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.072127 (+ 197us) server_negotiation.cc:407] Connection header received
0928 21:59:16.072190 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.072192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.072201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.072288 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.073278 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.073286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.073286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.073290 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.073330 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.073372 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.073442 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.073791 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.073807 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
I20260928 21:59:16.073989 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.071932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.2.165.67:43605)
0928 21:59:16.072089 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.072093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.072339 (+ 246us) server_negotiation.cc:407] Connection header received
0928 21:59:16.072366 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.072369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.072377 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.072440 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.073535 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.073544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.073545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.073549 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.073583 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.073609 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.073673 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.073875 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.073891 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260928 21:59:16.073817 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.071845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:43605 (local address 127.0.0.1:57952)
0928 21:59:16.072230 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.072238 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.072278 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.072721 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.072722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.072724 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:16.073433 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:16.073434 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:16.073440 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:16.073453 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:16.073687 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:16.073687 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:16.073688 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:59:16.073749 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:59:16.073762 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":66,"threads_started":1}
I20260928 21:59:16.073975 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.071609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35673 (local address 127.0.0.1:58506)
0928 21:59:16.072000 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.072045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.072126 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.072721 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.072721 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.072725 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:16.073564 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:16.073564 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:16.073570 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:16.073580 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:16.073850 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:16.073851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:16.073851 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 21:59:16.073894 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:59:16.073908 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":79,"threads_started":1}
I20260928 21:59:16.074195 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.071806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.2.165.65:35673)
0928 21:59:16.071976 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.071980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.072194 (+ 214us) server_negotiation.cc:407] Connection header received
0928 21:59:16.072226 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.072228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.072240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.072310 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.073702 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.073712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.073712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.073717 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.073752 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.073780 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.073827 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.073917 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.073933 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260928 21:59:16.074859 7446 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d700a2ee64ddfb15e72edf887dcc8 (DEFAULT_TABLE table=test-table [id=7d772a17068a49219d7d7caf79948602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:16.074860 7577 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d700a2ee64ddfb15e72edf887dcc8 (DEFAULT_TABLE table=test-table [id=7d772a17068a49219d7d7caf79948602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:16.075237 7577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d700a2ee64ddfb15e72edf887dcc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:16.075237 7446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d700a2ee64ddfb15e72edf887dcc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:16.078727 7789 tablet_bootstrap.cc:492] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Bootstrap starting.
I20260928 21:59:16.079166 7708 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d700a2ee64ddfb15e72edf887dcc8 (DEFAULT_TABLE table=test-table [id=7d772a17068a49219d7d7caf79948602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:16.079460 7789 tablet_bootstrap.cc:654] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:16.079478 7708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d700a2ee64ddfb15e72edf887dcc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:16.079754 7789 log.cc:824] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:16.080075 7788 tablet_bootstrap.cc:492] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb: Bootstrap starting.
I20260928 21:59:16.080619 7789 tablet_bootstrap.cc:492] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: No bootstrap required, opened a new log
I20260928 21:59:16.080724 7789 ts_tablet_manager.cc:1397] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:16.080787 7788 tablet_bootstrap.cc:654] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:16.081112 7788 log.cc:824] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:16.081929 7788 tablet_bootstrap.cc:492] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb: No bootstrap required, opened a new log
I20260928 21:59:16.082000 7788 ts_tablet_manager.cc:1397] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:59:16.082780 7792 tablet_bootstrap.cc:492] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Bootstrap starting.
I20260928 21:59:16.083524 7792 tablet_bootstrap.cc:654] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:16.083426 7789 raft_consensus.cc:348] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.083590 7789 raft_consensus.cc:374] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:16.083632 7789 raft_consensus.cc:729] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218a5b50734e4ef3a9d4540fd0462096, State: Initialized, Role: FOLLOWER
I20260928 21:59:16.083753 7789 consensus_queue.cc:260] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.083917 7792 log.cc:824] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:16.084029 7789 ts_tablet_manager.cc:1428] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:59:16.084169 7643 heartbeater.cc:503] Master 127.2.165.126:39381 was elected leader, sending a full tablet report...
I20260928 21:59:16.084725 7792 tablet_bootstrap.cc:492] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: No bootstrap required, opened a new log
I20260928 21:59:16.084789 7792 ts_tablet_manager.cc:1397] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:16.085078 7788 raft_consensus.cc:348] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.085383 7788 raft_consensus.cc:374] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:16.085428 7788 raft_consensus.cc:729] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a70a24b35b4647b5a738de820b5efb, State: Initialized, Role: FOLLOWER
I20260928 21:59:16.085546 7788 consensus_queue.cc:260] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.085865 7788 ts_tablet_manager.cc:1428] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 21:59:16.085908 7512 heartbeater.cc:503] Master 127.2.165.126:39381 was elected leader, sending a full tablet report...
I20260928 21:59:16.087276 7792 raft_consensus.cc:348] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.087396 7792 raft_consensus.cc:374] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:16.087417 7792 raft_consensus.cc:729] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7210e58da84687ad955bbb409db4a2, State: Initialized, Role: FOLLOWER
I20260928 21:59:16.087522 7792 consensus_queue.cc:260] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.087734 7792 ts_tablet_manager.cc:1428] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:59:16.087795 7774 heartbeater.cc:503] Master 127.2.165.126:39381 was elected leader, sending a full tablet report...
W20260928 21:59:16.192698 7644 tablet.cc:2369] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:16.289819 7795 raft_consensus.cc:482] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:16.289978 7795 raft_consensus.cc:504] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.290458 7795 leader_election.cc:288] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218a5b50734e4ef3a9d4540fd0462096 (127.2.165.66:37583), 0b7210e58da84687ad955bbb409db4a2 (127.2.165.67:43605)
I20260928 21:59:16.292028 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.290725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:37583 (local address 127.2.165.65:48021)
0928 21:59:16.290802 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.290816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.290861 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.291040 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.291041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.291044 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:16.291609 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:16.291610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:16.291619 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:16.291634 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:16.291873 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:16.291874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:16.291875 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:59:16.291935 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:59:16.291951 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 21:59:16.292070 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.290811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:48021 (local address 127.2.165.66:37583)
0928 21:59:16.290875 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.290878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.290890 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:59:16.290931 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.290932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.290939 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.291020 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.291742 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.291744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.291744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.291749 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.291772 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.291778 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.291844 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.291950 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.291979 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:59:16.292141 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.290889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:43605 (local address 127.2.165.65:56213)
0928 21:59:16.291084 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.291099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.291160 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.291340 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.291341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.291344 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:16.291799 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:16.291800 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:16.291806 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:16.291817 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:16.291993 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:16.291994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:16.291995 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:59:16.292046 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:59:16.292058 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":119,"threads_started":1}
I20260928 21:59:16.292137 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.290967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:56213 (local address 127.2.165.67:43605)
0928 21:59:16.291040 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.291042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.291153 (+ 111us) server_negotiation.cc:407] Connection header received
0928 21:59:16.291214 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.291215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.291220 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.291333 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.291898 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.291899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.291900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.291904 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.291924 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.291928 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.291984 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.292045 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.292059 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:59:16.292382 7597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d700a2ee64ddfb15e72edf887dcc8" candidate_uuid: "37a70a24b35b4647b5a738de820b5efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218a5b50734e4ef3a9d4540fd0462096" is_pre_election: true
I20260928 21:59:16.292460 7728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d700a2ee64ddfb15e72edf887dcc8" candidate_uuid: "37a70a24b35b4647b5a738de820b5efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7210e58da84687ad955bbb409db4a2" is_pre_election: true
I20260928 21:59:16.292547 7597 raft_consensus.cc:2454] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a70a24b35b4647b5a738de820b5efb in term 0.
I20260928 21:59:16.292620 7728 raft_consensus.cc:2454] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a70a24b35b4647b5a738de820b5efb in term 0.
I20260928 21:59:16.292793 7399 leader_election.cc:302] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218a5b50734e4ef3a9d4540fd0462096, 37a70a24b35b4647b5a738de820b5efb; no voters:
I20260928 21:59:16.292917 7795 raft_consensus.cc:2781] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:16.293013 7795 raft_consensus.cc:482] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:16.293051 7795 raft_consensus.cc:3037] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:16.293794 7795 raft_consensus.cc:504] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.293957 7795 leader_election.cc:288] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [CANDIDATE]: Term 1 election: Requested vote from peers 218a5b50734e4ef3a9d4540fd0462096 (127.2.165.66:37583), 0b7210e58da84687ad955bbb409db4a2 (127.2.165.67:43605)
I20260928 21:59:16.294188 7728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d700a2ee64ddfb15e72edf887dcc8" candidate_uuid: "37a70a24b35b4647b5a738de820b5efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7210e58da84687ad955bbb409db4a2"
I20260928 21:59:16.294168 7597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d700a2ee64ddfb15e72edf887dcc8" candidate_uuid: "37a70a24b35b4647b5a738de820b5efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218a5b50734e4ef3a9d4540fd0462096"
I20260928 21:59:16.294348 7728 raft_consensus.cc:3037] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:16.294391 7597 raft_consensus.cc:3037] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:16.295130 7728 raft_consensus.cc:2454] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a70a24b35b4647b5a738de820b5efb in term 1.
I20260928 21:59:16.295199 7597 raft_consensus.cc:2454] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a70a24b35b4647b5a738de820b5efb in term 1.
I20260928 21:59:16.295360 7399 leader_election.cc:302] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218a5b50734e4ef3a9d4540fd0462096, 37a70a24b35b4647b5a738de820b5efb; no voters:
I20260928 21:59:16.295495 7795 raft_consensus.cc:2781] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:16.295678 7795 raft_consensus.cc:686] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [term 1 LEADER]: Becoming Leader. State: Replica: 37a70a24b35b4647b5a738de820b5efb, State: Running, Role: LEADER
I20260928 21:59:16.295923 7795 consensus_queue.cc:237] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } }
I20260928 21:59:16.296902 7324 catalog_manager.cc:5495] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb reported cstate change: term changed from 0 to 1, leader changed from <none> to 37a70a24b35b4647b5a738de820b5efb (127.2.165.65). New cstate: current_term: 1 leader_uuid: "37a70a24b35b4647b5a738de820b5efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a70a24b35b4647b5a738de820b5efb" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 } health_report { overall_health: UNKNOWN } } }
W20260928 21:59:16.307751 7775 tablet.cc:2369] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:16.310673 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.309583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.2.165.65:35673)
0928 21:59:16.309642 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.309644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.309653 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:16.309699 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.309700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.309721 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.309785 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.310476 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.310478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.310479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.310483 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.310501 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.310505 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.310559 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.310605 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.310616 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:59:16.312757 7728 raft_consensus.cc:1261] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 1 FOLLOWER]: Refusing update from remote peer 37a70a24b35b4647b5a738de820b5efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:16.312759 7597 raft_consensus.cc:1261] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 1 FOLLOWER]: Refusing update from remote peer 37a70a24b35b4647b5a738de820b5efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:16.313107 7798 consensus_queue.cc:1035] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7210e58da84687ad955bbb409db4a2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:16.313251 7795 consensus_queue.cc:1035] T fb9d700a2ee64ddfb15e72edf887dcc8 P 37a70a24b35b4647b5a738de820b5efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "218a5b50734e4ef3a9d4540fd0462096" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:16.316658 7803 mvcc.cc:205] Tried to move back new op lower bound from 7334431769853624320 to 7334431769788764160. Current Snapshot: MvccSnapshot[applied={T|T < 7334431769853624320}]
I20260928 21:59:16.318452 7323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:16.318569 7323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:16.319456 7323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9369362222bb4dd080304b9f648c3f10: Sending DeleteTablet for 3 replicas of tablet fb9d700a2ee64ddfb15e72edf887dcc8
I20260928 21:59:16.319759 7446 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d700a2ee64ddfb15e72edf887dcc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:16 UTC) from {username='slave'} at 127.0.0.1:58506
I20260928 21:59:16.319851 7708 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d700a2ee64ddfb15e72edf887dcc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:16 UTC) from {username='slave'} at 127.0.0.1:57952
I20260928 21:59:16.320037 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 7384
I20260928 21:59:16.320128 7577 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d700a2ee64ddfb15e72edf887dcc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:16 UTC) from {username='slave'} at 127.0.0.1:49970
I20260928 21:59:16.322057 7811 tablet_replica.cc:330] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: stopping tablet replica
I20260928 21:59:16.322193 7811 raft_consensus.cc:2229] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:16.322252 7812 tablet_replica.cc:330] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: stopping tablet replica
I20260928 21:59:16.322333 7811 raft_consensus.cc:2258] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:16.322363 7812 raft_consensus.cc:2229] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:16.322463 7812 raft_consensus.cc:2258] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:16.322778 7811 ts_tablet_manager.cc:1905] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:16.322950 7812 ts_tablet_manager.cc:1905] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:16.324501 7812 ts_tablet_manager.cc:1918] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:16.324607 7812 log.cc:1196] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/fb9d700a2ee64ddfb15e72edf887dcc8
I20260928 21:59:16.324927 7812 ts_tablet_manager.cc:1939] T fb9d700a2ee64ddfb15e72edf887dcc8 P 218a5b50734e4ef3a9d4540fd0462096: Deleting consensus metadata
I20260928 21:59:16.324915 7811 ts_tablet_manager.cc:1918] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:16.325016 7811 log.cc:1196] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/fb9d700a2ee64ddfb15e72edf887dcc8
I20260928 21:59:16.325282 7811 ts_tablet_manager.cc:1939] T fb9d700a2ee64ddfb15e72edf887dcc8 P 0b7210e58da84687ad955bbb409db4a2: Deleting consensus metadata
I20260928 21:59:16.325537 7308 catalog_manager.cc:4841] TS 218a5b50734e4ef3a9d4540fd0462096 (127.2.165.66:37583): tablet fb9d700a2ee64ddfb15e72edf887dcc8 (table test-table [id=7d772a17068a49219d7d7caf79948602]) successfully deleted
I20260928 21:59:16.325933 7308 catalog_manager.cc:4841] TS 0b7210e58da84687ad955bbb409db4a2 (127.2.165.67:43605): tablet fb9d700a2ee64ddfb15e72edf887dcc8 (table test-table [id=7d772a17068a49219d7d7caf79948602]) successfully deleted
W20260928 21:59:16.326713 7310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:35673 (error 108)
I20260928 21:59:16.327255 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.327092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35673 (local address 127.0.0.1:58518)
0928 21:59:16.327157 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.327200 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:59:16.327338 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 7515
W20260928 21:59:16.328815 7310 catalog_manager.cc:4584] TS 37a70a24b35b4647b5a738de820b5efb (127.2.165.65:35673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb9d700a2ee64ddfb15e72edf887dcc8: Network error: Client connection negotiation failed: client connection to 127.2.165.65:35673: connect: Connection refused (error 111)
I20260928 21:59:16.332440 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 7646
I20260928 21:59:16.337543 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 7293
2026-09-28T21:59:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (818 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 21:59:16.357792 2709 mini_oidc.cc:152] Starting JWKS server
I20260928 21:59:16.358973 2709 webserver.cc:460] Webserver started at https://127.0.0.1:37745/ using document root <none> and password file <none>
I20260928 21:59:16.359112 2709 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:59:16.359308 2709 webserver.cc:460] Webserver started at http://127.0.0.1:34315/ using document root <none> and password file <none>
2026-09-28T21:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:16Z Disabled control of system clock
I20260928 21:59:16.375617 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:33529
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:33529
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37745/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/testchainca.cert with env {}
W20260928 21:59:16.460966 7819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:16.461184 7819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:16.461207 7819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:16.462380 7819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:16.462419 7819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:16.462431 7819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:16.462443 7819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:16.462456 7819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:59:16.462468 7819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:59:16.462479 7819 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37745/jwks.json
I20260928 21:59:16.463967 7819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:33529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:33529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37745/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:16.464160 7819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:16.464331 7819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:16.469401 7827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:16.469556 7824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:16.469398 7825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:16.469683 7819 server_base.cc:1034] running on GCE node
I20260928 21:59:16.469919 7819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:16.470193 7819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:16.471441 7819 hybrid_clock.cc:648] HybridClock initialized: now 1790632756471431 us; error 37 us; skew 500 ppm
I20260928 21:59:16.472875 7819 webserver.cc:460] Webserver started at http://127.2.165.126:35843/ using document root <none> and password file <none>
I20260928 21:59:16.473145 7819 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:16.473227 7819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:16.473371 7819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:16.474500 7819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "0a0df5e215424691ad477d0e47a47872"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "21ad2d60a53182a94383e0c396acaa56"
server_key_iv: "a2dfa9a30b5d0f51b8b9da19197788f1"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.474888 7819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "0a0df5e215424691ad477d0e47a47872"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "21ad2d60a53182a94383e0c396acaa56"
server_key_iv: "a2dfa9a30b5d0f51b8b9da19197788f1"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.476257 7819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:16.477094 7833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:16.477387 7819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:59:16.477511 7819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "0a0df5e215424691ad477d0e47a47872"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "21ad2d60a53182a94383e0c396acaa56"
server_key_iv: "a2dfa9a30b5d0f51b8b9da19197788f1"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.477600 7819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:16.495469 7819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:16.495846 7819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:16.496016 7819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:16.500267 7819 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:33529
I20260928 21:59:16.500306 7885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:33529 every 8 connection(s)
I20260928 21:59:16.500778 7819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:16.501780 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 7819
I20260928 21:59:16.501899 7886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:16.501914 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:16.502110 2709 external_mini_cluster.cc:1428] Setting key 0b87074a8f1ba88369a9cae9bc86807c
I20260928 21:59:16.505270 7886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872: Bootstrap starting.
I20260928 21:59:16.506016 7886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:16.506321 7886 log.cc:824] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:16.507038 7886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872: No bootstrap required, opened a new log
I20260928 21:59:16.508085 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.502660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.2.165.126:33529)
0928 21:59:16.502862 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.502870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.502896 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:59:16.503501 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.503509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.503576 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.503693 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.503930 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.504881 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.505951 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.506506 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.506639 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:16.507400 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.507404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.507405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.507582 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.507615 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.507626 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.507697 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.507816 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.507923 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260928 21:59:16.509204 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:59:16.509342 7886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0df5e215424691ad477d0e47a47872" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 33529 } }
I20260928 21:59:16.509508 7886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:16.509531 7886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0df5e215424691ad477d0e47a47872, State: Initialized, Role: FOLLOWER
I20260928 21:59:16.509673 7886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0df5e215424691ad477d0e47a47872" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 33529 } }
I20260928 21:59:16.509744 7886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:16.509766 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:16.509832 7886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:16.510562 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0df5e215424691ad477d0e47a47872" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 33529 } }
I20260928 21:59:16.510681 7886 leader_election.cc:302] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a0df5e215424691ad477d0e47a47872; no voters:
I20260928 21:59:16.510848 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:16.510912 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:16.511055 7891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [term 1 LEADER]: Becoming Leader. State: Replica: 0a0df5e215424691ad477d0e47a47872, State: Running, Role: LEADER
I20260928 21:59:16.511118 7886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:16.511144 7891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0df5e215424691ad477d0e47a47872" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 33529 } }
I20260928 21:59:16.511531 7893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0df5e215424691ad477d0e47a47872. Latest consensus state: current_term: 1 leader_uuid: "0a0df5e215424691ad477d0e47a47872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0df5e215424691ad477d0e47a47872" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 33529 } } }
I20260928 21:59:16.511521 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a0df5e215424691ad477d0e47a47872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0df5e215424691ad477d0e47a47872" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 33529 } } }
I20260928 21:59:16.511626 7893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:16.511634 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872 [sys.catalog]: This master's current role is: LEADER
W20260928 21:59:16.512987 7907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:59:16.513056 7907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:59:16.513140 7898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:16.513339 7898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:16.515045 7898 catalog_manager.cc:1347] Generated new cluster ID: 9c293d6caaf14d8c99205fc3c0f5c5c3
I20260928 21:59:16.515108 7898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:16.534307 7898 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:16.535074 7898 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:16.546558 7898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a0df5e215424691ad477d0e47a47872: Generated new TSK 0
I20260928 21:59:16.546818 7898 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:59:16.579368 7916 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 21:59:16.579509 7916 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:59:16.579783 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.563474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.2.165.126:33529)
0928 21:59:16.563531 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.563534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.563684 (+ 150us) server_negotiation.cc:407] Connection header received
0928 21:59:16.563804 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.563805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.563819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.563912 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:59:16.564148 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.565008 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.565916 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.566234 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.566308 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:16.566561 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:59:16.579485 (+ 12924us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:59:16.579708 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 21:59:16.579845 7889 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 21:59:16.580330 7915 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:16.563301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:33529 (local address 127.0.0.1:47908)
0928 21:59:16.563605 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.563617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.563705 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.563983 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.563984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.563987 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:59:16.564075 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:16.564085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.565135 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.565136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:16.565822 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:16.565824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.566331 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.566332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:16.566401 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:16.566429 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:59:16.579721 (+ 13292us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:59:16.579926 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.165.126:33529: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.165.126:33529: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 21:59:16.580844 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 7819
2026-09-28T21:59:16Z chronyd exiting
I20260928 21:59:16.617424 2709 test_util.cc:175] -----------------------------------------------
I20260928 21:59:16.617537 2709 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632730261496-2709-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (261 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 21:59:16.618402 2709 mini_oidc.cc:152] Starting JWKS server
I20260928 21:59:16.619138 2709 webserver.cc:460] Webserver started at https://127.0.0.1:37417/ using document root <none> and password file <none>
I20260928 21:59:16.619207 2709 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:59:16.619395 2709 webserver.cc:460] Webserver started at http://127.0.0.1:44193/ using document root <none> and password file <none>
2026-09-28T21:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:16Z Disabled control of system clock
I20260928 21:59:16.627130 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45233
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45233
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37417/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/testchainca.cert with env {}
W20260928 21:59:16.711123 7922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:16.711429 7922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:16.711476 7922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:16.712838 7922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:16.712924 7922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:16.712945 7922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:16.712966 7922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:16.712996 7922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:59:16.713025 7922 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:59:16.713052 7922 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37417/jwks.json
I20260928 21:59:16.714775 7922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37417/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:16.715093 7922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:16.715348 7922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:59:16.720937 7922 server_base.cc:1034] running on GCE node
W20260928 21:59:16.720917 7930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:16.720920 7928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:16.721072 7927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:16.721431 7922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:16.721769 7922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:16.722977 7922 hybrid_clock.cc:648] HybridClock initialized: now 1790632756722968 us; error 42 us; skew 500 ppm
I20260928 21:59:16.729099 7922 webserver.cc:460] Webserver started at http://127.2.165.126:40463/ using document root <none> and password file <none>
I20260928 21:59:16.729422 7922 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:16.729506 7922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:16.729668 7922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:16.730861 7922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "4f488192474c4d08ba57d3c26622553a"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "5d14b98fae2deb35fa7845e4688c4955"
server_key_iv: "e7e61ad6015f8af936ce70f2b51a5cf3"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.731249 7922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "4f488192474c4d08ba57d3c26622553a"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "5d14b98fae2deb35fa7845e4688c4955"
server_key_iv: "e7e61ad6015f8af936ce70f2b51a5cf3"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.732743 7922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:16.733723 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:16.733951 7922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:16.734076 7922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "4f488192474c4d08ba57d3c26622553a"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "5d14b98fae2deb35fa7845e4688c4955"
server_key_iv: "e7e61ad6015f8af936ce70f2b51a5cf3"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.734203 7922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:16.754253 7922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:16.754626 7922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:16.754788 7922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:16.759218 7922 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45233
I20260928 21:59:16.759281 7988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45233 every 8 connection(s)
I20260928 21:59:16.759667 7922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:16.760639 7989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:16.762282 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 7922
I20260928 21:59:16.762435 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:16.762672 2709 external_mini_cluster.cc:1428] Setting key 773e93a58407c11fd0526fce42a6637f
I20260928 21:59:16.763361 7989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a: Bootstrap starting.
I20260928 21:59:16.763952 7989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:16.764254 7989 log.cc:824] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:16.765125 7989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a: No bootstrap required, opened a new log
I20260928 21:59:16.767287 7989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488192474c4d08ba57d3c26622553a" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45233 } }
I20260928 21:59:16.767450 7989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:16.767498 7989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f488192474c4d08ba57d3c26622553a, State: Initialized, Role: FOLLOWER
I20260928 21:59:16.767652 7989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488192474c4d08ba57d3c26622553a" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45233 } }
I20260928 21:59:16.767740 7989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:16.767783 7989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:16.767833 7989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:16.768391 7989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488192474c4d08ba57d3c26622553a" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45233 } }
I20260928 21:59:16.768510 7989 leader_election.cc:302] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f488192474c4d08ba57d3c26622553a; no voters:
I20260928 21:59:16.768740 7989 leader_election.cc:288] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:16.768740 7994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:16.768895 7994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [term 1 LEADER]: Becoming Leader. State: Replica: 4f488192474c4d08ba57d3c26622553a, State: Running, Role: LEADER
I20260928 21:59:16.768978 7994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488192474c4d08ba57d3c26622553a" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45233 } }
I20260928 21:59:16.769040 7989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:16.769337 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f488192474c4d08ba57d3c26622553a. Latest consensus state: current_term: 1 leader_uuid: "4f488192474c4d08ba57d3c26622553a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488192474c4d08ba57d3c26622553a" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45233 } } }
I20260928 21:59:16.769426 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:16.769618 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.763327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.2.165.126:45233)
0928 21:59:16.763600 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.763609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.763639 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:59:16.764310 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.764319 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.764439 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.764649 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.764957 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.765923 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.767210 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.767789 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.767928 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:16.768601 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.768605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.768606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.768889 (+ 283us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.768939 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.768956 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.769276 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.769306 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.769392 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
I20260928 21:59:16.769594 7996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f488192474c4d08ba57d3c26622553a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488192474c4d08ba57d3c26622553a" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45233 } } }
I20260928 21:59:16.769680 7996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:16.769961 7999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:16.770452 7999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:16.770771 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:59:16.772394 7999 catalog_manager.cc:1347] Generated new cluster ID: 555914aa2c73497f8f9b61a78b4cf307
I20260928 21:59:16.772454 7999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:16.777647 7999 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:16.778237 7999 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:16.784104 7999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f488192474c4d08ba57d3c26622553a: Generated new TSK 0
I20260928 21:59:16.784313 7999 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:59:16.807446 8019 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 21:59:16.807546 8019 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:59:16.807957 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.793768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.2.165.126:45233)
0928 21:59:16.793836 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.793838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.793930 (+ 92us) server_negotiation.cc:407] Connection header received
0928 21:59:16.794032 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.794033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.794045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.794145 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:59:16.794288 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.794811 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.795611 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.795831 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.795904 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:16.796098 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:59:16.807695 (+ 11597us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37417/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:59:16.807870 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37417/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:59:16.808024 7992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37417/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 21:59:16.808054 8018 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:16.793664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45233 (local address 127.0.0.1:58628)
0928 21:59:16.793844 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:16.793855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:16.793944 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:16.794147 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:16.794148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:16.794149 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:59:16.794213 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:16.794215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.794930 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.794931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:16.795522 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:16.795523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.795928 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.795929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:16.795983 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:16.795988 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:59:16.807831 (+ 11843us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37417/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:59:16.807947 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.165.126:45233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37417/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260928 21:59:16.808386 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 7922
2026-09-28T21:59:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 21:59:16.845842 2709 mini_oidc.cc:152] Starting JWKS server
I20260928 21:59:16.846513 2709 webserver.cc:460] Webserver started at https://127.0.0.1:40725/ using document root <none> and password file <none>
I20260928 21:59:16.846555 2709 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:59:16.846693 2709 webserver.cc:460] Webserver started at http://127.0.0.1:45889/ using document root <none> and password file <none>
2026-09-28T21:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:16Z Disabled control of system clock
I20260928 21:59:16.855050 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45683
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45683
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40725/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:59:16.937427 8025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:16.937678 8025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:16.937716 8025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:16.938901 8025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:16.938968 8025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:16.938989 8025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:16.939011 8025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:16.939043 8025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:59:16.939066 8025 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:59:16.939085 8025 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40725/jwks.json
I20260928 21:59:16.940450 8025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40725/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:16.940752 8025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:16.941051 8025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:16.946362 8031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:16.946403 8033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:16.946362 8030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:16.946717 8025 server_base.cc:1034] running on GCE node
I20260928 21:59:16.946979 8025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:16.947348 8025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:16.948522 8025 hybrid_clock.cc:648] HybridClock initialized: now 1790632756948499 us; error 42 us; skew 500 ppm
I20260928 21:59:16.950062 8025 webserver.cc:460] Webserver started at http://127.2.165.126:44325/ using document root <none> and password file <none>
I20260928 21:59:16.950387 8025 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:16.950474 8025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:16.950660 8025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:16.951942 8025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "b446e51135d64b25b18d13fffeaca149"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "07223a2f2a7e02fcb2d939177957b97d"
server_key_iv: "1a710eb9a8514d7e9b1631d873082767"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.952361 8025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "b446e51135d64b25b18d13fffeaca149"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "07223a2f2a7e02fcb2d939177957b97d"
server_key_iv: "1a710eb9a8514d7e9b1631d873082767"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.954010 8025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:16.954936 8039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:16.955152 8025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:16.955274 8025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "b446e51135d64b25b18d13fffeaca149"
format_stamp: "Formatted at 2026-09-28 21:59:16 on dist-test-slave-3t36"
server_key: "07223a2f2a7e02fcb2d939177957b97d"
server_key_iv: "1a710eb9a8514d7e9b1631d873082767"
server_key_version: "encryptionkey@0"
I20260928 21:59:16.955397 8025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:16.975345 8025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:16.975726 8025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:16.975898 8025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:16.980118 8025 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45683
I20260928 21:59:16.980206 8091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45683 every 8 connection(s)
I20260928 21:59:16.980567 8025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:16.981380 8092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:16.984344 8092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149: Bootstrap starting.
I20260928 21:59:16.984951 8092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:16.985219 8092 log.cc:824] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:16.986153 8092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149: No bootstrap required, opened a new log
I20260928 21:59:16.987797 8092 raft_consensus.cc:348] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b446e51135d64b25b18d13fffeaca149" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45683 } }
I20260928 21:59:16.987946 8092 raft_consensus.cc:374] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:16.987970 8092 raft_consensus.cc:729] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b446e51135d64b25b18d13fffeaca149, State: Initialized, Role: FOLLOWER
I20260928 21:59:16.988083 8092 consensus_queue.cc:260] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b446e51135d64b25b18d13fffeaca149" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45683 } }
I20260928 21:59:16.988137 8092 raft_consensus.cc:388] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:16.988157 8092 raft_consensus.cc:482] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:16.988185 8092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:16.988806 8092 raft_consensus.cc:504] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b446e51135d64b25b18d13fffeaca149" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45683 } }
I20260928 21:59:16.988899 8092 leader_election.cc:302] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b446e51135d64b25b18d13fffeaca149; no voters:
I20260928 21:59:16.989029 8092 leader_election.cc:288] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:16.989099 8095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:16.989269 8095 raft_consensus.cc:686] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [term 1 LEADER]: Becoming Leader. State: Replica: b446e51135d64b25b18d13fffeaca149, State: Running, Role: LEADER
I20260928 21:59:16.989356 8092 sys_catalog.cc:564] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:16.989373 8095 consensus_queue.cc:237] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b446e51135d64b25b18d13fffeaca149" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45683 } }
I20260928 21:59:16.989744 8096 sys_catalog.cc:455] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b446e51135d64b25b18d13fffeaca149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b446e51135d64b25b18d13fffeaca149" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45683 } } }
I20260928 21:59:16.989831 8096 sys_catalog.cc:458] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:16.989916 8097 sys_catalog.cc:455] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b446e51135d64b25b18d13fffeaca149. Latest consensus state: current_term: 1 leader_uuid: "b446e51135d64b25b18d13fffeaca149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b446e51135d64b25b18d13fffeaca149" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45683 } } }
I20260928 21:59:16.989976 8097 sys_catalog.cc:458] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:16.990139 8101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:16.990128 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8025
I20260928 21:59:16.990231 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:16.990408 2709 external_mini_cluster.cc:1428] Setting key 2d081005005428d698f3133d537d9357
I20260928 21:59:16.990614 8101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:16.992659 8101 catalog_manager.cc:1347] Generated new cluster ID: 8dd03b564ab8432ba1adb9c70a017871
I20260928 21:59:16.992740 8101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:16.996052 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:16.990937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.2.165.126:45683)
0928 21:59:16.991432 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:16.991440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:16.991464 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:59:16.991698 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:16.991705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:16.991780 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:16.991928 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:16.992175 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.993284 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.994134 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:16.994552 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:16.994687 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:16.995385 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:16.995389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:16.995390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:16.995569 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:16.995603 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:16.995613 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:16.995692 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:16.995767 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:59:16.995850 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":70,"threads_started":1}
I20260928 21:59:16.997098 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:59:17.036823 8101 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:17.037570 8101 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:17.043004 8101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b446e51135d64b25b18d13fffeaca149: Generated new TSK 0
I20260928 21:59:17.043238 8101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:59:17.065505 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:17.061885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.2.165.126:45683)
0928 21:59:17.061958 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:17.061961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:17.062095 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:59:17.062220 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:17.062221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:17.062233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:17.062317 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:59:17.062474 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.062976 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.063838 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.064032 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.064108 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:17.064317 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:59:17.065273 (+ 956us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:59:17.065412 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 21:59:17.065580 8113 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 21:59:17.065646 8121 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:17.061813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45683 (local address 127.0.0.1:50636)
0928 21:59:17.062015 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:17.062024 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:17.062101 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:17.062318 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:17.062319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:17.062321 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:59:17.062384 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:17.062389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.063067 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.063068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:17.063731 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:17.063733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.064114 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.064116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:17.064185 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:17.064191 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:59:17.065396 (+ 1205us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:59:17.065531 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.165.126:45683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40725/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20260928 21:59:17.067464 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8025
E20260928 21:59:17.076344 8122 webserver.cc:533] Webserver: sslize failed:
E20260928 21:59:17.076426 8122 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T21:59:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (259 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T21:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:17Z Disabled control of system clock
I20260928 21:59:17.113020 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41663
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:41663
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/testchainca.cert with env {}
W20260928 21:59:17.194922 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:17.195156 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:17.195180 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:17.196337 8126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:17.196377 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:17.196390 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:17.196403 8126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:17.196415 8126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:59:17.196427 8126 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:59:17.196439 8126 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 21:59:17.198048 8126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:41663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:17.198280 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:17.198448 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:17.206498 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:17.206696 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:17.206575 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:17.207479 8126 server_base.cc:1034] running on GCE node
I20260928 21:59:17.207803 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:17.208218 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:17.209396 8126 hybrid_clock.cc:648] HybridClock initialized: now 1790632757209371 us; error 38 us; skew 500 ppm
I20260928 21:59:17.210629 8126 webserver.cc:460] Webserver started at http://127.2.165.126:45807/ using document root <none> and password file <none>
I20260928 21:59:17.210844 8126 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:17.210888 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:17.210979 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:17.212046 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "85cf5d2884704449904327e66473a9c8"
format_stamp: "Formatted at 2026-09-28 21:59:17 on dist-test-slave-3t36"
server_key: "f06dd527bdc89d2fce10c5db7417ca16"
server_key_iv: "dd013a5d8957514b7e70d3678ed3dd64"
server_key_version: "encryptionkey@0"
I20260928 21:59:17.212355 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "85cf5d2884704449904327e66473a9c8"
format_stamp: "Formatted at 2026-09-28 21:59:17 on dist-test-slave-3t36"
server_key: "f06dd527bdc89d2fce10c5db7417ca16"
server_key_iv: "dd013a5d8957514b7e70d3678ed3dd64"
server_key_version: "encryptionkey@0"
I20260928 21:59:17.213838 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:17.214684 8140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:17.214881 8126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:17.214991 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "85cf5d2884704449904327e66473a9c8"
format_stamp: "Formatted at 2026-09-28 21:59:17 on dist-test-slave-3t36"
server_key: "f06dd527bdc89d2fce10c5db7417ca16"
server_key_iv: "dd013a5d8957514b7e70d3678ed3dd64"
server_key_version: "encryptionkey@0"
I20260928 21:59:17.215083 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:17.224077 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:17.224404 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:17.224563 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:17.228472 8126 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:41663
I20260928 21:59:17.228545 8192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:41663 every 8 connection(s)
I20260928 21:59:17.228947 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:17.229976 8193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:17.232556 8193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8: Bootstrap starting.
I20260928 21:59:17.233240 8193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:17.233508 8193 log.cc:824] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:17.234207 8193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8: No bootstrap required, opened a new log
I20260928 21:59:17.235948 8193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cf5d2884704449904327e66473a9c8" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41663 } }
I20260928 21:59:17.236135 8193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:17.236162 8193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cf5d2884704449904327e66473a9c8, State: Initialized, Role: FOLLOWER
I20260928 21:59:17.236322 8193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cf5d2884704449904327e66473a9c8" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41663 } }
I20260928 21:59:17.236421 8193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:17.236450 8193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:17.236481 8193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:17.237103 8193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cf5d2884704449904327e66473a9c8" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41663 } }
I20260928 21:59:17.237201 8193 leader_election.cc:302] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85cf5d2884704449904327e66473a9c8; no voters:
I20260928 21:59:17.237387 8193 leader_election.cc:288] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:17.237425 8196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:17.237537 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8126
I20260928 21:59:17.237592 8196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [term 1 LEADER]: Becoming Leader. State: Replica: 85cf5d2884704449904327e66473a9c8, State: Running, Role: LEADER
I20260928 21:59:17.237638 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:17.237699 8193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:17.237854 2709 external_mini_cluster.cc:1428] Setting key da47ff0d97e2b705e43aeff15e3de03c
I20260928 21:59:17.237869 8196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cf5d2884704449904327e66473a9c8" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41663 } }
I20260928 21:59:17.238207 8196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85cf5d2884704449904327e66473a9c8. Latest consensus state: current_term: 1 leader_uuid: "85cf5d2884704449904327e66473a9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cf5d2884704449904327e66473a9c8" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41663 } } }
I20260928 21:59:17.238332 8196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:17.238642 8198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85cf5d2884704449904327e66473a9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cf5d2884704449904327e66473a9c8" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41663 } } }
I20260928 21:59:17.238783 8198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:17.239230 8205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:17.240000 8205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:17.241629 8205 catalog_manager.cc:1347] Generated new cluster ID: 07c450f425ed44f1aab9dd2548f8a8aa
I20260928 21:59:17.241688 8205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:17.244982 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:17.238403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.2.165.126:41663)
0928 21:59:17.238603 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:17.238615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:17.238858 (+ 243us) server_negotiation.cc:407] Connection header received
0928 21:59:17.239081 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:17.239091 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:17.239168 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:17.239317 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:17.239699 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.241145 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.242266 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.242899 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.243014 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:17.244019 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:17.244023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:17.244024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:17.244237 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:17.244281 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:17.244296 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:17.244356 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:17.244639 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 21:59:17.244753 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
I20260928 21:59:17.246116 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:59:17.253176 8205 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:17.253778 8205 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:17.256362 8205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85cf5d2884704449904327e66473a9c8: Generated new TSK 0
I20260928 21:59:17.256558 8205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:59:17.745149 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:17.262510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.2.165.126:41663)
0928 21:59:17.262558 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:17.262560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:17.262668 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:59:17.262727 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:17.262728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:17.262741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:17.262833 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:59:17.262980 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.263747 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.264637 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.264998 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.265071 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:17.265251 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:59:17.744907 (+479656us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:59:17.745083 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55308: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 21:59:17.745246 8207 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55308: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 21:59:17.745291 8222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:17.262408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:41663 (local address 127.0.0.1:55308)
0928 21:59:17.262582 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:17.262595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:17.262666 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:17.262844 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:17.262845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:17.262847 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:59:17.262909 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:17.262911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.263866 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.263866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:17.264545 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:17.264547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:17.265086 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:17.265087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:17.265135 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:17.265141 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:59:17.745066 (+479925us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:59:17.745206 (+ 140us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.165.126:41663: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":120,"threads_started":1}
I20260928 21:59:17.745936 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8126
2026-09-28T21:59:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (658 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:18 dist-test-slave-3t36 krb5kdc[8234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:18 dist-test-slave-3t36 krb5kdc[8234](info): set up 2 sockets
Sep 28 21:59:18 dist-test-slave-3t36 krb5kdc[8234](info): commencing operation
krb5kdc: starting...
W20260928 21:59:19.005002 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:19.128773 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38079
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:19.210652 8247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:19.210944 8247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:19.210985 8247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:19.212272 8247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:19.212368 8247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:19.212393 8247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:19.212414 8247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:19.212442 8247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:19.214099 8247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:19.214408 8247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:19.214654 8247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:19.219743 8253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:19.219882 8247 server_base.cc:1034] running on GCE node
W20260928 21:59:19.219789 8252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:19.219978 8255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:19.220249 8247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:19.220616 8247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:19.221850 8247 hybrid_clock.cc:648] HybridClock initialized: now 1790632759221831 us; error 52 us; skew 500 ppm
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.224507 8247 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:19.225119 8247 webserver.cc:460] Webserver started at http://127.2.165.126:36229/ using document root <none> and password file <none>
I20260928 21:59:19.225386 8247 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:19.225463 8247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:19.225626 8247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:19.226747 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "dad1056715514f1f95e3dfb0491af78f"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "03bc002963f4220c8b498a42f4f6b3f1"
server_key_iv: "99c59a29826f34021b2bccd89a5ccd13"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.227133 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "dad1056715514f1f95e3dfb0491af78f"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "03bc002963f4220c8b498a42f4f6b3f1"
server_key_iv: "99c59a29826f34021b2bccd89a5ccd13"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.228592 8247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:19.229543 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.229766 8247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:19.229883 8247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "dad1056715514f1f95e3dfb0491af78f"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "03bc002963f4220c8b498a42f4f6b3f1"
server_key_iv: "99c59a29826f34021b2bccd89a5ccd13"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.229961 8247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:19.241745 8247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:19.244226 8247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:19.244403 8247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:19.249150 8247 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:38079
I20260928 21:59:19.249184 8314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:38079 every 8 connection(s)
I20260928 21:59:19.249603 8247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:19.250662 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:19.253005 8315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f: Bootstrap starting.
I20260928 21:59:19.253566 8315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:19.253818 8315 log.cc:824] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:19.254099 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8247
I20260928 21:59:19.254216 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:19.254416 2709 external_mini_cluster.cc:1428] Setting key 29962a0349de0826a163a068dedc99db
I20260928 21:59:19.254670 8315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f: No bootstrap required, opened a new log
I20260928 21:59:19.256529 8315 raft_consensus.cc:348] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad1056715514f1f95e3dfb0491af78f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38079 } }
I20260928 21:59:19.256721 8315 raft_consensus.cc:374] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:19.256812 8315 raft_consensus.cc:729] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad1056715514f1f95e3dfb0491af78f, State: Initialized, Role: FOLLOWER
I20260928 21:59:19.256994 8315 consensus_queue.cc:260] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad1056715514f1f95e3dfb0491af78f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38079 } }
I20260928 21:59:19.257090 8315 raft_consensus.cc:388] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:19.257130 8315 raft_consensus.cc:482] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:19.257182 8315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:19.258270 8315 raft_consensus.cc:504] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad1056715514f1f95e3dfb0491af78f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38079 } }
I20260928 21:59:19.258400 8315 leader_election.cc:302] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dad1056715514f1f95e3dfb0491af78f; no voters:
I20260928 21:59:19.258580 8315 leader_election.cc:288] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:19.258618 8320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:19.258771 8320 raft_consensus.cc:686] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [term 1 LEADER]: Becoming Leader. State: Replica: dad1056715514f1f95e3dfb0491af78f, State: Running, Role: LEADER
I20260928 21:59:19.258845 8315 sys_catalog.cc:564] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:19.258921 8320 consensus_queue.cc:237] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad1056715514f1f95e3dfb0491af78f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38079 } }
I20260928 21:59:19.259392 8321 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dad1056715514f1f95e3dfb0491af78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad1056715514f1f95e3dfb0491af78f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38079 } } }
I20260928 21:59:19.259469 8321 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:19.259658 8322 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dad1056715514f1f95e3dfb0491af78f. Latest consensus state: current_term: 1 leader_uuid: "dad1056715514f1f95e3dfb0491af78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad1056715514f1f95e3dfb0491af78f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38079 } } }
I20260928 21:59:19.259752 8322 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:19.260124 8328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:19.260655 8328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:19.262331 8328 catalog_manager.cc:1347] Generated new cluster ID: 0c4a45a614104ea88148c4b0b0bb0c74
I20260928 21:59:19.262384 8328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:19.267392 8328 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:19.267894 8328 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:19.268045 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.255043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.2.165.126:38079)
0928 21:59:19.255239 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.255249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.255288 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:59:19.256516 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.256525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.256591 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.256829 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.257646 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.258820 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.259936 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.260532 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.260645 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.263617 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:19.263621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:19.263621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:19.263634 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:19.265781 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.266205 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.266209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.266209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.266338 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.266551 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.266552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.266553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.266887 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:19.267578 (+ 691us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:19.267655 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:59:19.267803 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260928 21:59:19.274160 8328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dad1056715514f1f95e3dfb0491af78f: Generated new TSK 0
I20260928 21:59:19.274371 8328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:59:19.307955 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38079
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:19.393890 8341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:19.394117 8341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:19.394142 8341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:19.395308 8341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:19.395350 8341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:19.395378 8341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:19.396908 8341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:19.397113 8341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:19.397332 8341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:19.402777 8347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:19.402714 8346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:19.402700 8349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:19.402851 8341 server_base.cc:1034] running on GCE node
I20260928 21:59:19.403246 8341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:19.403541 8341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:19.404784 8341 hybrid_clock.cc:648] HybridClock initialized: now 1790632759404771 us; error 31 us; skew 500 ppm
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.407454 8341 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:59:19.408004 8341 webserver.cc:460] Webserver started at http://127.2.165.65:35889/ using document root <none> and password file <none>
I20260928 21:59:19.408244 8341 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:19.408314 8341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:19.408470 8341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:19.409598 8341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "e3453044c5b64d59898d65865899ecd1"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "2ea5197a7fa7a28f9ac8f107756e4f2a"
server_key_iv: "fe84971b324cc6c92116c4174c11db37"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.409976 8341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "e3453044c5b64d59898d65865899ecd1"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "2ea5197a7fa7a28f9ac8f107756e4f2a"
server_key_iv: "fe84971b324cc6c92116c4174c11db37"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.411319 8341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:19.412199 8356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.412617 8341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:19.412766 8341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "e3453044c5b64d59898d65865899ecd1"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "2ea5197a7fa7a28f9ac8f107756e4f2a"
server_key_iv: "fe84971b324cc6c92116c4174c11db37"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.412863 8341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:19.438839 8341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:19.441679 8341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:19.441880 8341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:19.442186 8341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:19.442550 8341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:19.442607 8341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.442642 8341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:19.442672 8341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.448913 8341 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:33533
I20260928 21:59:19.448971 8469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:33533 every 8 connection(s)
I20260928 21:59:19.449355 8341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:19.453631 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8341
I20260928 21:59:19.453747 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:19.453940 2709 external_mini_cluster.cc:1428] Setting key 048f3350558d88a5b0e2db2d5f446500
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:19.461153 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.450313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:34205 (local address 127.2.165.126:38079)
0928 21:59:19.450366 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.450368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.450622 (+ 254us) server_negotiation.cc:407] Connection header received
0928 21:59:19.451671 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.451673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.451688 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.451756 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.452529 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.453070 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.454613 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.454883 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.454946 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.457694 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:19.457696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:19.457696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:19.457705 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:19.459519 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.459982 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.459985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.459986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.460055 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.460325 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.460327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.460327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.460464 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:19.460541 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:19.460958 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 21:59:19.461054 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:59:19.461392 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.450195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:38079 (local address 127.2.165.65:34205)
0928 21:59:19.450430 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:19.450479 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:19.451555 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:19.451853 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:19.451857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:19.452144 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:19.452406 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:19.452410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.453182 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.453183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:19.454449 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:19.454452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.454964 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.454965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:19.455138 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.455896 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:19.455908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:19.457566 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:19.459609 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:19.459619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:19.459622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:19.459895 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:19.460124 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:19.460124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:19.460125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:19.460252 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:19.460531 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:19.460578 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:19.460810 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 21:59:19.460932 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:59:19.461113 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
I20260928 21:59:19.461944 8470 heartbeater.cc:347] Connected to a master server at 127.2.165.126:38079
I20260928 21:59:19.462071 8470 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:19.462273 8470 heartbeater.cc:511] Master 127.2.165.126:38079 requested a full tablet report, sending...
I20260928 21:59:19.462898 8279 ts_manager.cc:194] Registered new tserver with Master: e3453044c5b64d59898d65865899ecd1 (127.2.165.65:33533)
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
I20260928 21:59:19.463712 8279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:34205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:59:19.485172 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38079
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:19.568806 8475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:19.569082 8475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:19.569125 8475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:19.570406 8475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:19.570483 8475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:19.570533 8475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:19.572293 8475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:19.572571 8475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:19.572901 8475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:19.578915 8480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:19.578917 8483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:19.578962 8481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:19.579154 8475 server_base.cc:1034] running on GCE node
I20260928 21:59:19.579416 8475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:19.579722 8475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:19.580868 8475 hybrid_clock.cc:648] HybridClock initialized: now 1790632759580860 us; error 36 us; skew 500 ppm
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.583702 8475 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:59:19.584211 8475 webserver.cc:460] Webserver started at http://127.2.165.66:37561/ using document root <none> and password file <none>
I20260928 21:59:19.584463 8475 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:19.584535 8475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:19.584719 8475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:19.585825 8475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "2f0fcab609ad4e3f8d18701073265305"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "ff409b73f70b0798624b867872f1fd8e"
server_key_iv: "ed05d5cfb69ae48eae4fea2d3720a603"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.586210 8475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "2f0fcab609ad4e3f8d18701073265305"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "ff409b73f70b0798624b867872f1fd8e"
server_key_iv: "ed05d5cfb69ae48eae4fea2d3720a603"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.587682 8475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:19.588505 8490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.588740 8475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:19.588842 8475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "2f0fcab609ad4e3f8d18701073265305"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "ff409b73f70b0798624b867872f1fd8e"
server_key_iv: "ed05d5cfb69ae48eae4fea2d3720a603"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.588949 8475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:19.607868 8475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:19.610365 8475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:19.610536 8475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:19.610790 8475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:19.611120 8475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:19.611176 8475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.611218 8475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:19.611250 8475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.617487 8475 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:35923
I20260928 21:59:19.617537 8603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:35923 every 8 connection(s)
I20260928 21:59:19.617894 8475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:19.620199 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8475
I20260928 21:59:19.620323 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:19.620496 2709 external_mini_cluster.cc:1428] Setting key d56ab159dd212db24861ac5258dbd7a4
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 21:59:19.628188 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.618850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:55333 (local address 127.2.165.126:38079)
0928 21:59:19.618956 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.618959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.619129 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:59:19.620188 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.620189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.620202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.620262 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.621079 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.621645 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.622671 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.622871 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.622967 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.625258 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:19.625260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:19.625260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:19.625268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:19.626730 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.627187 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.627188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.627189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.627251 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.627515 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.627517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.627517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.627642 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:19.627718 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:19.627998 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 21:59:19.628090 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:59:19.628393 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.618753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:38079 (local address 127.2.165.66:55333)
0928 21:59:19.618991 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:19.619039 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:19.620070 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:19.620343 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:19.620347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:19.620747 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:19.620985 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:19.620991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.621746 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.621747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:19.622577 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:19.622580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.622974 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.622975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:19.623105 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.623659 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:19.623668 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:19.625151 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:19.626851 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:19.626858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:19.626861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:19.627095 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:19.627325 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:19.627326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:19.627326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:19.627446 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:19.627709 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:19.627715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:19.627867 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 21:59:19.627979 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:59:19.628156 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260928 21:59:19.628911 8604 heartbeater.cc:347] Connected to a master server at 127.2.165.126:38079
I20260928 21:59:19.629021 8604 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:19.629220 8604 heartbeater.cc:511] Master 127.2.165.126:38079 requested a full tablet report, sending...
I20260928 21:59:19.629642 8279 ts_manager.cc:194] Registered new tserver with Master: 2f0fcab609ad4e3f8d18701073265305 (127.2.165.66:35923)
I20260928 21:59:19.630044 8279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:55333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:59:19.648967 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38079
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:19.732760 8609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:19.733011 8609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:19.733039 8609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:19.734434 8609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:19.734490 8609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:19.734520 8609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:19.736167 8609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:19.736443 8609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:19.736713 8609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:19.742213 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:19.742228 8614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:19.742220 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:19.742380 8609 server_base.cc:1034] running on GCE node
I20260928 21:59:19.742652 8609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:19.742913 8609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:19.744093 8609 hybrid_clock.cc:648] HybridClock initialized: now 1790632759744086 us; error 51 us; skew 500 ppm
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.746618 8609 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:59:19.747062 8609 webserver.cc:460] Webserver started at http://127.2.165.67:45317/ using document root <none> and password file <none>
I20260928 21:59:19.747296 8609 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:19.747342 8609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:19.747484 8609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:19.748553 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "0e161265b0044d3bb056c4de27390b56"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "1d01e8412212794613b7dd314bedd2f8"
server_key_iv: "a1e41450d7843ae3127f8bc74dade0b4"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.748989 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "0e161265b0044d3bb056c4de27390b56"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "1d01e8412212794613b7dd314bedd2f8"
server_key_iv: "a1e41450d7843ae3127f8bc74dade0b4"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.750520 8609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:19.751483 8624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.751791 8609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:19.751896 8609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "0e161265b0044d3bb056c4de27390b56"
format_stamp: "Formatted at 2026-09-28 21:59:19 on dist-test-slave-3t36"
server_key: "1d01e8412212794613b7dd314bedd2f8"
server_key_iv: "a1e41450d7843ae3127f8bc74dade0b4"
server_key_version: "encryptionkey@0"
I20260928 21:59:19.751953 8609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:19.769621 8609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:19.771679 8609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:19.771796 8609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:19.772014 8609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:19.772363 8609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:19.772431 8609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.772477 8609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:19.772502 8609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:19.778934 8609 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:43535
I20260928 21:59:19.778981 8737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:43535 every 8 connection(s)
I20260928 21:59:19.779304 8609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:19.784559 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8609
I20260928 21:59:19.784708 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:19.784909 2709 external_mini_cluster.cc:1428] Setting key 372bc26b0838536c399df71b61c7f8d2
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:19.789181 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.780053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:40229 (local address 127.2.165.126:38079)
0928 21:59:19.780110 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.780112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.780316 (+ 204us) server_negotiation.cc:407] Connection header received
0928 21:59:19.781344 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.781346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.781360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.781417 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.782065 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.782560 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.783530 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.783717 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.783773 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.786453 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:19.786454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:19.786454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:19.786461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:19.787859 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.788259 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.788260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.788261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.788320 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.788545 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.788546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.788546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.788650 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:19.788798 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:19.789037 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 21:59:19.789117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:59:19.789342 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.780035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:38079 (local address 127.2.165.67:40229)
0928 21:59:19.780186 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:19.780224 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:19.781244 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:19.781482 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:19.781486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:19.781757 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:19.781974 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:19.781978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.782634 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.782635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:19.783452 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:19.783455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.783789 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.783790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:19.783886 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.784703 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:19.784713 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:19.786369 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:19.787930 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:19.787935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:19.787937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:19.788142 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:19.788394 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:19.788394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:19.788395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:19.788488 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:19.788784 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:19.788787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:19.788904 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 21:59:19.788987 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:59:19.789106 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260928 21:59:19.789880 8738 heartbeater.cc:347] Connected to a master server at 127.2.165.126:38079
I20260928 21:59:19.789961 8738 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:19.790096 8738 heartbeater.cc:511] Master 127.2.165.126:38079 requested a full tablet report, sending...
I20260928 21:59:19.790413 8279 ts_manager.cc:194] Registered new tserver with Master: 0e161265b0044d3bb056c4de27390b56 (127.2.165.67:43535)
I20260928 21:59:19.790755 8279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:40229
I20260928 21:59:19.797186 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/test-admin.keytab.
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.813658 2709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 21:59:19.815641 8745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:19.814711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.2.165.65:33533)
0928 21:59:19.814917 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.814923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.814935 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:59:19.814967 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.814969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.814986 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.815077 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.815555 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58466: BlockingRecv error: recv got EOF from 127.0.0.1:58466 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
W20260928 21:59:19.815797 2709 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.817163 2709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.818167 2709 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:59:19.826187 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.818734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.2.165.65:33533)
0928 21:59:19.818798 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.818799 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.818809 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:19.818912 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.818913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.818924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.818980 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.819351 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.820078 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.820837 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.821188 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.821250 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.823209 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:19.823216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:19.823217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:19.823223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:19.824844 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.825216 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.825219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.825219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.825285 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.825509 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.825510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.825510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.825649 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:19.825755 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:19.825959 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 21:59:19.826025 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:59:19.828420 8745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:19.827676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.2.165.65:33533)
0928 21:59:19.827723 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.827725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.827737 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:59:19.827855 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.827856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.827874 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.827944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.828280 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58488: BlockingRecv error: recv got EOF from 127.0.0.1:58488 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 21:59:19.828523 2709 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.829643 2709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.830607 2709 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:59:19.838431 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.831203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.2.165.65:33533)
0928 21:59:19.831241 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.831243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.831252 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:19.831333 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.831334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.831390 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.831451 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.831847 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.832363 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.833169 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.833378 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.833464 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.835644 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:19.835645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:19.835645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:19.835652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:19.837221 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.837558 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.837559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.837559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.837642 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.837866 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.837868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.837868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.838013 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:19.838101 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:19.838281 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 21:59:19.838360 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:59:19.840289 8745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:19.839751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.2.165.65:33533)
0928 21:59:19.839789 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.839791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.839816 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:59:19.839835 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.839836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.839846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.839910 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.840232 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58506: BlockingRecv error: recv got EOF from 127.0.0.1:58506 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:59:19.840437 2709 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632730261496-2709-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.841529 2709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:19.842530 2709 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 21:59:19.850155 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:19.843169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.2.165.65:33533)
0928 21:59:19.843244 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:19.843248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:19.843366 (+ 118us) server_negotiation.cc:407] Connection header received
0928 21:59:19.843432 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:19.843433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:19.843441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:19.843489 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:19.843833 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.844309 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.845113 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:19.845300 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:19.845371 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:19.847583 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:19.847584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:19.847585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:19.847591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:19.849116 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.849444 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.849445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.849446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.849499 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:19.849677 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:19.849677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:19.849677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:19.849771 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:19.849847 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:19.849949 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 21:59:19.850008 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:59:19.850880 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8341
I20260928 21:59:19.855989 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8475
I20260928 21:59:19.860901 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8609
I20260928 21:59:19.865547 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8247
2026-09-28T21:59:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1938 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8758](info): set up 2 sockets
Sep 28 21:59:19 dist-test-slave-3t36 krb5kdc[8758](info): commencing operation
krb5kdc: starting...
W20260928 21:59:20.913058 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:20 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab.
Entry for principal oryx/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab.
I20260928 21:59:21.041208 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:34319
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab
--principal=oryx/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:21.123812 8771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:21.124022 8771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:21.124045 8771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:21.125276 8771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:21.125324 8771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:21.125339 8771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:21.125351 8771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:21.125406 8771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:21.126955 8771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:34319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab
--principal=oryx/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:21.127228 8771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:21.127465 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:21.132483 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.132539 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.132717 8771 server_base.cc:1034] running on GCE node
W20260928 21:59:21.132725 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.133074 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:21.133431 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:21.134580 8771 hybrid_clock.cc:648] HybridClock initialized: now 1790632761134561 us; error 39 us; skew 500 ppm
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:21.137285 8771 init.cc:377] Logged in from keytab as oryx/127.2.165.126@KRBTEST.COM (short username oryx)
I20260928 21:59:21.137861 8771 webserver.cc:460] Webserver started at http://127.2.165.126:36233/ using document root <none> and password file <none>
I20260928 21:59:21.138121 8771 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:21.138168 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:21.138357 8771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:21.139487 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "4cbf333743754a5f80314e9e2eae266f"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "d13615c6becfce9fc7886dce62fef46b"
server_key_iv: "7dedc182f5869e08792e16e61318c417"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.139879 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "4cbf333743754a5f80314e9e2eae266f"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "d13615c6becfce9fc7886dce62fef46b"
server_key_iv: "7dedc182f5869e08792e16e61318c417"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.141461 8771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:21.142323 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.142563 8771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:21.142685 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "4cbf333743754a5f80314e9e2eae266f"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "d13615c6becfce9fc7886dce62fef46b"
server_key_iv: "7dedc182f5869e08792e16e61318c417"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.142772 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:21.152124 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:21.154630 8771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:21.154821 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:21.159204 8771 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:34319
I20260928 21:59:21.159232 8838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:34319 every 8 connection(s)
I20260928 21:59:21.159622 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:21.160638 8839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:21.163034 8839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: Bootstrap starting.
I20260928 21:59:21.163642 8839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:21.163942 8839 log.cc:824] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:21.164906 8839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: No bootstrap required, opened a new log
I20260928 21:59:21.166440 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8771
I20260928 21:59:21.166568 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:21.166765 2709 external_mini_cluster.cc:1428] Setting key fb1c3fec94e5e4b5eda247e448d4de41
I20260928 21:59:21.167215 8839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } }
I20260928 21:59:21.167465 8839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:21.167552 8839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbf333743754a5f80314e9e2eae266f, State: Initialized, Role: FOLLOWER
I20260928 21:59:21.167719 8839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } }
I20260928 21:59:21.167810 8839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:21.167845 8839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:21.167898 8839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:21.169171 8839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } }
I20260928 21:59:21.169297 8839 leader_election.cc:302] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbf333743754a5f80314e9e2eae266f; no voters:
I20260928 21:59:21.169464 8839 leader_election.cc:288] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:21.169528 8844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:21.169726 8844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 1 LEADER]: Becoming Leader. State: Replica: 4cbf333743754a5f80314e9e2eae266f, State: Running, Role: LEADER
I20260928 21:59:21.169744 8839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:21.169879 8844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } }
I20260928 21:59:21.170517 8845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbf333743754a5f80314e9e2eae266f. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } }
I20260928 21:59:21.170953 8845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:21.170876 8846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } }
I20260928 21:59:21.171203 8846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:21.171394 8857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:21.172102 8857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:21.173909 8857 catalog_manager.cc:1347] Generated new cluster ID: b3f89f07593f487d8dd67a400cac3541
I20260928 21:59:21.173962 8857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:21.179656 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.167432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.2.165.126:34319)
0928 21:59:21.167650 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.167661 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.167696 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:59:21.167825 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.167834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.167897 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.168056 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.168702 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.170284 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.171471 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.172096 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.172230 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.175364 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.175368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.175368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.175381 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.177458 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.177905 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.177908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.177909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.178043 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.178279 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.178280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.178280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.178585 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.179230 (+ 645us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.179302 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.179444 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260928 21:59:21.215909 8857 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:21.216815 8857 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:21.221292 8857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: Generated new TSK 0
I20260928 21:59:21.221515 8857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab.
Entry for principal oryx/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab.
I20260928 21:59:21.248947 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34319
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab
--principal=oryx/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:21.332062 8865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:21.332273 8865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:21.332299 8865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:21.333801 8865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:21.333856 8865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:21.333891 8865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:21.335345 8865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab
--principal=oryx/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:21.335547 8865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:21.335738 8865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:21.341065 8870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.341082 8871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.341298 8865 server_base.cc:1034] running on GCE node
W20260928 21:59:21.341078 8873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.341589 8865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:21.341809 8865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:21.342954 8865 hybrid_clock.cc:648] HybridClock initialized: now 1790632761342933 us; error 43 us; skew 500 ppm
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:21.345817 8865 init.cc:377] Logged in from keytab as oryx/127.2.165.65@KRBTEST.COM (short username oryx)
I20260928 21:59:21.346309 8865 webserver.cc:460] Webserver started at http://127.2.165.65:39183/ using document root <none> and password file <none>
I20260928 21:59:21.346576 8865 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:21.346625 8865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:21.346810 8865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:21.347954 8865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "fc9c2ec29eb540fbae683aa694dd6c6b"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "b5642463da4c7331c2c175c7a80e99b8"
server_key_iv: "e59803b38a6b58559ffa29a75b4dba78"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.348331 8865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "fc9c2ec29eb540fbae683aa694dd6c6b"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "b5642463da4c7331c2c175c7a80e99b8"
server_key_iv: "e59803b38a6b58559ffa29a75b4dba78"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.350011 8865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:21.350973 8880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.351179 8865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:21.351269 8865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "fc9c2ec29eb540fbae683aa694dd6c6b"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "b5642463da4c7331c2c175c7a80e99b8"
server_key_iv: "e59803b38a6b58559ffa29a75b4dba78"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.351321 8865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:21.359920 8865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:21.362406 8865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:21.362529 8865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:21.362748 8865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:21.363082 8865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:21.363119 8865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.363142 8865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:21.363155 8865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.369125 8865 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:40909
I20260928 21:59:21.369199 8993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:40909 every 8 connection(s)
I20260928 21:59:21.369591 8865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:21.373965 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8865
I20260928 21:59:21.374090 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:21.374253 2709 external_mini_cluster.cc:1428] Setting key 9f4e0e49f066591be8eb5fed8224b392
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:21.380352 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.370479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:51445 (local address 127.2.165.126:34319)
0928 21:59:21.370571 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.370574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.370770 (+ 196us) server_negotiation.cc:407] Connection header received
0928 21:59:21.371342 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.371344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.371360 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.371423 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.372461 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.373076 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.374124 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.374343 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.374436 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.377108 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.377110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.377110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.377120 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.378851 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.379255 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.379256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.379257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.379320 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.379561 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.379562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.379562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.379725 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.379805 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.380113 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.380214 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:59:21.380364 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.370435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:51445)
0928 21:59:21.370643 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.370686 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.371232 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.371501 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.371504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.372140 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.372349 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.372354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.373171 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.373172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.374026 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.374028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.374447 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.374449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.374601 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.375246 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.375254 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.377011 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.378950 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.378955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.378957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.379177 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.379392 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.379393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.379393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.379494 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.379795 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.379799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.379918 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.380052 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.380196 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260928 21:59:21.380981 8994 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34319
I20260928 21:59:21.381102 8994 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:21.381305 8994 heartbeater.cc:511] Master 127.2.165.126:34319 requested a full tablet report, sending...
I20260928 21:59:21.381945 8803 ts_manager.cc:194] Registered new tserver with Master: fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
I20260928 21:59:21.382889 8803 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:51445
WARNING: no policy specified for oryx/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab.
Entry for principal oryx/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab.
I20260928 21:59:21.403564 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34319
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab
--principal=oryx/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:21.488027 8999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:21.488283 8999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:21.488327 8999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:21.489573 8999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:21.489648 8999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:21.489694 8999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:21.491250 8999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab
--principal=oryx/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:21.491544 8999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:21.491797 8999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:21.497105 9005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.497117 9007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.497241 8999 server_base.cc:1034] running on GCE node
W20260928 21:59:21.497143 9004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.497575 8999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:21.497874 8999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:21.499042 8999 hybrid_clock.cc:648] HybridClock initialized: now 1790632761499030 us; error 31 us; skew 500 ppm
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:21.501808 8999 init.cc:377] Logged in from keytab as oryx/127.2.165.66@KRBTEST.COM (short username oryx)
I20260928 21:59:21.502421 8999 webserver.cc:460] Webserver started at http://127.2.165.66:41211/ using document root <none> and password file <none>
I20260928 21:59:21.502679 8999 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:21.502748 8999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:21.502903 8999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:21.503968 8999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "0bf075309fb547d08fce67d52f331086"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "a5316d2c38dfd948e9bb70588d8fec43"
server_key_iv: "0c6844fcedce9a60d68a5969b0c359bf"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.504321 8999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "0bf075309fb547d08fce67d52f331086"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "a5316d2c38dfd948e9bb70588d8fec43"
server_key_iv: "0c6844fcedce9a60d68a5969b0c359bf"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.505791 8999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:21.506583 9014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.506775 8999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:21.506880 8999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "0bf075309fb547d08fce67d52f331086"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "a5316d2c38dfd948e9bb70588d8fec43"
server_key_iv: "0c6844fcedce9a60d68a5969b0c359bf"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.506958 8999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:21.522401 8999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:21.524823 8999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:21.525028 8999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:21.525291 8999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:21.525633 8999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:21.525688 8999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.525724 8999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:21.525774 8999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.531898 8999 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:38935
I20260928 21:59:21.531936 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:38935 every 8 connection(s)
I20260928 21:59:21.532292 8999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:21.539131 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 8999
I20260928 21:59:21.539247 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:21.539459 2709 external_mini_cluster.cc:1428] Setting key 8f1b470612f5f362c3915a72a7a5c669
I20260928 21:59:21.542516 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.533111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:60631 (local address 127.2.165.126:34319)
0928 21:59:21.533193 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.533196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.533387 (+ 191us) server_negotiation.cc:407] Connection header received
0928 21:59:21.533854 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.533855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.533866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.533933 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.534919 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.535438 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.536452 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.536739 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.536852 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.539214 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.539216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.539216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.539224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.540971 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.541428 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.541430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.541430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.541502 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.541811 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.541812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.541812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.541938 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.541999 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.542371 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.542450 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:59:21.542796 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.533039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:60631)
0928 21:59:21.533196 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.533240 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.533733 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.534023 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.534027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.534598 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.534795 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.534800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.535522 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.535523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.536348 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.536350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.536858 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.536858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.536970 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.537590 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.537599 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.539105 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.541069 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.541078 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.541081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.541311 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.541591 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.541592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.541592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.541737 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.542016 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.542021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.542204 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.542326 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.542527 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260928 21:59:21.543337 9128 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34319
I20260928 21:59:21.543489 9128 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:21.543697 9128 heartbeater.cc:511] Master 127.2.165.126:34319 requested a full tablet report, sending...
I20260928 21:59:21.544145 8803 ts_manager.cc:194] Registered new tserver with Master: 0bf075309fb547d08fce67d52f331086 (127.2.165.66:38935)
I20260928 21:59:21.544554 8803 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:60631
WARNING: no policy specified for oryx/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab.
Entry for principal oryx/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab.
I20260928 21:59:21.568343 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34319
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab
--principal=oryx/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:21.651834 9133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:21.652084 9133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:21.652125 9133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:21.653441 9133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:21.653518 9133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:21.653558 9133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:21.655121 9133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab
--principal=oryx/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:21.655324 9133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:21.655520 9133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:21.660629 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.660624 9139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.660624 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.660996 9133 server_base.cc:1034] running on GCE node
I20260928 21:59:21.661233 9133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:21.661540 9133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:21.662730 9133 hybrid_clock.cc:648] HybridClock initialized: now 1790632761662721 us; error 38 us; skew 500 ppm
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:21.665549 9133 init.cc:377] Logged in from keytab as oryx/127.2.165.67@KRBTEST.COM (short username oryx)
I20260928 21:59:21.666069 9133 webserver.cc:460] Webserver started at http://127.2.165.67:45243/ using document root <none> and password file <none>
I20260928 21:59:21.666332 9133 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:21.666402 9133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:21.666553 9133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:21.667631 9133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "8ecd339997536dd178c939b2c040287c"
server_key_iv: "c34d2c1e004896d04394fa4053eeb3b0"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.667994 9133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "8ecd339997536dd178c939b2c040287c"
server_key_iv: "c34d2c1e004896d04394fa4053eeb3b0"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.669552 9133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:59:21.670400 9148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.670702 9133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:21.670804 9133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "8ecd339997536dd178c939b2c040287c"
server_key_iv: "c34d2c1e004896d04394fa4053eeb3b0"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.670939 9133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:21.696313 9133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:21.698462 9133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:21.698581 9133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:21.698766 9133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:21.699093 9133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:21.699128 9133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.699151 9133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:21.699200 9133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.705266 9133 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:42575
I20260928 21:59:21.705329 9261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:42575 every 8 connection(s)
I20260928 21:59:21.705672 9133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:21.714174 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 9133
I20260928 21:59:21.714308 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:21.714473 2709 external_mini_cluster.cc:1428] Setting key a4e719b3bd7947fb52e31398ea6a0256
I20260928 21:59:21.716171 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.706493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:50759 (local address 127.2.165.126:34319)
0928 21:59:21.706563 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.706565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.706800 (+ 235us) server_negotiation.cc:407] Connection header received
0928 21:59:21.707263 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.707264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.707273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.707386 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.708253 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.708879 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.709866 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.710143 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.710201 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.712721 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.712723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.712723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.712732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.714228 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.715005 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.715007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.715007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.715085 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.715412 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.715413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.715414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.715583 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.715670 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.715977 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.716070 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:59:21.716460 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.706473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:50759)
0928 21:59:21.706671 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.706708 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.707180 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.707383 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.707386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.707965 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.708163 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.708168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.708955 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.708956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.709767 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.709769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.710217 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.710218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.710365 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.710948 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.710956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.712543 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.714372 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.714381 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.714384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.714862 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.715192 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.715193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.715193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.715324 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.715659 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.715666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.715823 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.715953 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.716183 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260928 21:59:21.717052 9262 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34319
I20260928 21:59:21.717156 9262 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:21.717387 9262 heartbeater.cc:511] Master 127.2.165.126:34319 requested a full tablet report, sending...
I20260928 21:59:21.717830 8803 ts_manager.cc:194] Registered new tserver with Master: db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67:42575)
I20260928 21:59:21.718185 8803 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:50759
I20260928 21:59:21.726986 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.165.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.125.keytab.
Entry for principal oryx/127.2.165.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.125.keytab.
I20260928 21:59:21.755796 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:33015
--webserver_interface=127.2.165.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.125.keytab
--principal=oryx/127.2.165.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:34319,127.2.165.125:33015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:21.838598 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:21.838830 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:21.838853 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:21.840081 9267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:21.840134 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:21.840148 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:21.840161 9267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:21.840173 9267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:21.841931 9267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:34319,127.2.165.125:33015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.125.keytab
--principal=oryx/127.2.165.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.165.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
W20260928 21:59:21.842046 9267 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.165.126:34319,127.2.165.125:33015'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 21:59:21.842238 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:21.842494 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:21.847607 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.847615 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.847630 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.847761 9267 server_base.cc:1034] running on GCE node
I20260928 21:59:21.848086 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:21.848410 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:21.849581 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790632761849497 us; error 133 us; skew 500 ppm
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:21.859089 9267 init.cc:377] Logged in from keytab as oryx/127.2.165.125@KRBTEST.COM (short username oryx)
I20260928 21:59:21.859622 9267 webserver.cc:460] Webserver started at http://127.2.165.125:38941/ using document root <none> and password file <none>
I20260928 21:59:21.859894 9267 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:21.859972 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:21.860121 9267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:21.861305 9267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/instance:
uuid: "e0c923da36c948d39913ba44657480cd"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "92958ab5c41276e2b1e1ec7263a6e5bf"
server_key_iv: "2c8976e849e15f44d05e522ad5804e63"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.861682 9267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal/instance:
uuid: "e0c923da36c948d39913ba44657480cd"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "92958ab5c41276e2b1e1ec7263a6e5bf"
server_key_iv: "2c8976e849e15f44d05e522ad5804e63"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.863050 9267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:21.863848 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.864059 9267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:21.864168 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
uuid: "e0c923da36c948d39913ba44657480cd"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "92958ab5c41276e2b1e1ec7263a6e5bf"
server_key_iv: "2c8976e849e15f44d05e522ad5804e63"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.864245 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:21.885140 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:21.887614 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:21.887815 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:21.892016 9267 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.125:33015
I20260928 21:59:21.892050 9334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.125:33015 every 8 connection(s)
I20260928 21:59:21.892431 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
I20260928 21:59:21.893543 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:21.895227 9335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:59:21.901321 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 9267
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.125@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:21.905573 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.895945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.2.165.126:34319)
0928 21:59:21.896011 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.896013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.896206 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:59:21.896755 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.896757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.896767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.896830 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.897477 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.898002 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.899063 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.899246 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.899305 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.902073 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.902075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.902075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.902085 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.903920 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.904329 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.904330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.904331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.904401 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.904790 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.904791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.904792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.904968 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.905043 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.905385 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.905478 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:59:21.905689 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.895892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53138)
0928 21:59:21.896069 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.896112 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.896609 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.896905 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.896909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.897154 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.897394 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.897399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.898089 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.898090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.898977 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.898980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.899353 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.899354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.899518 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.900087 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.900096 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.901976 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.904013 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.904019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.904021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.904252 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.904516 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.904517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.904517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.904663 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.905045 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.905050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.905217 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.905324 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.905478 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260928 21:59:21.906529 9335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.165.125@KRBTEST.COM
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.125@KRBTEST.COM for oryx/127.2.165.125@KRBTEST.COM
I20260928 21:59:21.912127 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.901914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.2.165.125:33015)
0928 21:59:21.902138 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.902143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.902166 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:59:21.902187 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.902189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.902204 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.902284 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.903135 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.904323 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.905296 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.905876 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.905964 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.908635 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.908642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.908643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.908650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.910707 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.911110 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.911114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.911115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.911179 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.911513 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.911514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.911514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.911684 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.911860 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.911927 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.912008 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":121,"threads_started":1}
I20260928 21:59:21.914667 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.906855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.0.0.1:42810)
0928 21:59:21.906906 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.906917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.906960 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.907205 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.907206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.907354 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.907438 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.907440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.908082 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.908083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.908629 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.908631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.909180 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.909182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.909264 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.909745 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.909749 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.911286 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.913644 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.913645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.913646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.913806 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.914031 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.914032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.914032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.914086 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.914341 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.914341 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.914379 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.914446 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.914548 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:59:21.914750 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.906924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.2.165.125:33015)
0928 21:59:21.907104 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.907105 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.907110 (+ 5us) server_negotiation.cc:407] Connection header received
0928 21:59:21.907124 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.907125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.907132 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.907204 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.907522 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.908004 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.908783 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.909089 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.909159 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.911372 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.911373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.911374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.911380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.913547 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.913889 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.913890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.913890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.913951 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.914152 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.914153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.914153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.914274 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.914351 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.914473 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.914599 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260928 21:59:21.916126 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Bootstrap starting.
I20260928 21:59:21.916852 9335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:21.917151 9335 log.cc:824] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:21.918123 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: No bootstrap required, opened a new log
I20260928 21:59:21.919688 9335 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } }
I20260928 21:59:21.919828 9335 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:21.919852 9335 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c923da36c948d39913ba44657480cd, State: Initialized, Role: FOLLOWER
I20260928 21:59:21.920003 9335 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } }
I20260928 21:59:21.920357 9335 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:21.920328 9342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } } }
I20260928 21:59:21.920594 9342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:59:21.922075 9353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:59:21.922165 9353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:59:21.923987 9267 master_runner.cc:428] Detected that this master e0c923da36c948d39913ba44657480cd is joining an existing cluster
I20260928 21:59:21.924063 9267 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.165.125:33015
I20260928 21:59:21.931838 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.925365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.2.165.125:33015)
0928 21:59:21.925407 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.925409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.925532 (+ 123us) server_negotiation.cc:407] Connection header received
0928 21:59:21.925617 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.925618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.925635 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.925662 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.926082 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.926775 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.927538 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.927769 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.927856 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.928982 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.928984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.928984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.928991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.930670 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.931065 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.931067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.931067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.931119 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.931321 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.931322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.931323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.931442 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.931498 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.931677 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.931759 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:59:21.932113 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.925151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.2.165.126:34319)
0928 21:59:21.925206 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.925209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.925352 (+ 143us) server_negotiation.cc:407] Connection header received
0928 21:59:21.925440 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.925441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.925453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.925561 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.925925 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.926727 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.927890 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.928191 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.928233 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.929275 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.929277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.929277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.929284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.930833 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.931235 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.931236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.931236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.931288 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.931511 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.931512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.931512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.931651 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.931715 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.931920 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.932010 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:59:21.932297 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.925098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53140)
0928 21:59:21.925269 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.925279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.925380 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.925568 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.925569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.925788 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.925847 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.925849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.926819 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.926821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.927784 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.927787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.928310 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.928311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.928373 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.928805 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.928810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.929197 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.930920 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.930921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.930922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.931130 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.931354 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.931355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.931355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.931432 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.931728 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.931730 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.931787 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.931882 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.932174 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":42,"thread_start_us":92,"threads_started":1}
I20260928 21:59:21.932536 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.925319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.0.0.1:42812)
0928 21:59:21.925471 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.925479 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.925544 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.925714 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.925715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.925939 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.926002 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.926004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.926850 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.926850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.927451 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.927453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.927863 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.927864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.927918 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.928418 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.928423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.928885 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.930759 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.930760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.930761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.930996 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.931191 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.931192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.931193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.931249 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.931541 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.931542 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.931577 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.932106 (+ 529us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.932432 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":271,"thread_start_us":101,"threads_started":1}
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.126@KRBTEST.COM for oryx/127.2.165.125@KRBTEST.COM
I20260928 21:59:21.944164 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.934663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.0.0.1:42826)
0928 21:59:21.936822 (+ 2159us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.936861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.936950 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.937121 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.937127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.937410 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.937467 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.937469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.938126 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.938127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.938745 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.938747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.939159 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.939160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.939249 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.939756 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.939761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.941377 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.943167 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.943173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.943175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.943367 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.943571 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.943572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.943572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.943625 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.943882 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.943888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.943931 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.943998 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.944092 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2133,"thread_start_us":81,"threads_started":1}
I20260928 21:59:21.944214 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.934719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.2.165.125:33015)
0928 21:59:21.934753 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.934755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.936924 (+ 2169us) server_negotiation.cc:407] Connection header received
0928 21:59:21.937034 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.937035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.937045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.937117 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.937572 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.938038 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.938842 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.939075 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.939156 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.941464 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.941466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.941466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.941473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.943085 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.943437 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.943438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.943438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.943497 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.943687 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.943688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.943688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.943820 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.943888 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.944041 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.944121 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:59:21.945125 8803 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.165.125:33015: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e0c923da36c948d39913ba44657480cd"
member_type: NON_VOTER
last_known_addr {
host: "127.2.165.125"
port: 33015
}
attrs {
promote: true
}
}
dest_uuid: "4cbf333743754a5f80314e9e2eae266f"
cas_config_opid_index: -1
I20260928 21:59:21.945513 8803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } }
I20260928 21:59:21.946014 9363 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e0c923da36c948d39913ba44657480cd (127.2.165.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } }
I20260928 21:59:21.946305 9309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:21.947144 9363 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.165.125:33015
I20260928 21:59:21.947157 9309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Refusing update from remote peer 4cbf333743754a5f80314e9e2eae266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 21:59:21.947402 9267 master.cc:537] Master@127.2.165.125:33015 shutting down...
I20260928 21:59:21.947463 9363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:21.947597 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } }
I20260928 21:59:21.947718 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
W20260928 21:59:21.948287 8790 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 21:59:21.948803 8790 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f -> Peer e0c923da36c948d39913ba44657480cd (127.2.165.125:33015): Couldn't send request to peer e0c923da36c948d39913ba44657480cd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:59:21.950320 9267 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:21.950415 9267 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:21.950466 9267 tablet_replica.cc:330] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: stopping tablet replica
I20260928 21:59:21.963687 9267 master.cc:552] Master@127.2.165.125:33015 shutdown complete.
I20260928 21:59:21.963757 9267 master_runner.cc:305] Clearing existing system tablet
I20260928 21:59:21.963894 9267 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:21.963943 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:21.964707 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:21.965245 9367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.965366 9267 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:21.965427 9267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.965466 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
uuid: "e0c923da36c948d39913ba44657480cd"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "92958ab5c41276e2b1e1ec7263a6e5bf"
server_key_iv: "2c8976e849e15f44d05e522ad5804e63"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.966279 9267 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:21.967949 9267 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:59:21.968037 9267 log.cc:1196] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 21:59:21.968245 9267 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Deleting consensus metadata
I20260928 21:59:21.968401 9267 master_runner.cc:315] Copying system tablet from 127.2.165.126:34319
I20260928 21:59:21.969202 9267 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: tablet copy: Beginning tablet copy session from remote peer at address 127.2.165.126:34319
I20260928 21:59:21.975813 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.969616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.2.165.126:34319)
0928 21:59:21.969672 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:21.969674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:21.969741 (+ 67us) server_negotiation.cc:407] Connection header received
0928 21:59:21.969825 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:21.969826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:21.969836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:21.969903 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:21.970274 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.970784 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.971706 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.971925 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.972013 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.973086 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:21.973087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:21.973088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:21.973095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:21.974623 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.974995 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.974997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.974997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.975062 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:21.975263 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:21.975264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:21.975264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:21.975390 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:21.975463 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:21.975632 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 21:59:21.975713 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:59:21.975828 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:21.969542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53156)
0928 21:59:21.969670 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:21.969684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:21.969759 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:21.969910 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:21.969911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:21.970099 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:21.970199 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.970201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.970870 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.970871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.971621 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:21.971623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:21.972015 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:21.972016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:21.972087 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:21.972598 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:21.972603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:21.972976 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:21.974715 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.974716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.974717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.974919 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.975129 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:21.975130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:21.975131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:21.975190 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:21.975465 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:21.975466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:21.975514 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:59:21.975596 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:59:21.975714 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":83,"threads_started":1}
I20260928 21:59:21.976145 8823 tablet_copy_service.cc:140] P 4cbf333743754a5f80314e9e2eae266f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e0c923da36c948d39913ba44657480cd ({username='oryx', principal='oryx/127.2.165.125@KRBTEST.COM'} at 127.0.0.1:53156)
I20260928 21:59:21.976250 8823 tablet_copy_service.cc:161] P 4cbf333743754a5f80314e9e2eae266f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e0c923da36c948d39913ba44657480cd at {username='oryx', principal='oryx/127.2.165.125@KRBTEST.COM'} at 127.0.0.1:53156: session id = e0c923da36c948d39913ba44657480cd-00000000000000000000000000000000
I20260928 21:59:21.976985 8823 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:59:21.977634 9267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:59:21.978978 9267 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: tablet copy: Starting download of 0 data blocks...
I20260928 21:59:21.979143 9267 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: tablet copy: Starting download of 1 WAL segments...
I20260928 21:59:21.979960 9267 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:59:21.981053 8823 tablet_copy_service.cc:342] P 4cbf333743754a5f80314e9e2eae266f: Request end of tablet copy session e0c923da36c948d39913ba44657480cd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.165.125@KRBTEST.COM'} at 127.0.0.1:53156
I20260928 21:59:21.981143 8823 tablet_copy_service.cc:433] P 4cbf333743754a5f80314e9e2eae266f: ending tablet copy session e0c923da36c948d39913ba44657480cd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e0c923da36c948d39913ba44657480cd
I20260928 21:59:21.983706 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:21.985206 9374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.985316 9377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:21.985334 9375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:21.988876 9267 server_base.cc:1034] running on GCE node
I20260928 21:59:21.989001 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:21.989159 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:21.990301 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790632761990285 us; error 38 us; skew 500 ppm
Sep 28 21:59:21 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:21.991725 9267 init.cc:377] Logged in from keytab as oryx/127.2.165.125@KRBTEST.COM (short username oryx)
I20260928 21:59:21.992069 9267 webserver.cc:460] Webserver started at http://127.2.165.125:33875/ using document root <none> and password file <none>
I20260928 21:59:21.992216 9267 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:21.992262 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:21.993121 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:21.993741 9384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:21.993928 9267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:21.993983 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
uuid: "e0c923da36c948d39913ba44657480cd"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "92958ab5c41276e2b1e1ec7263a6e5bf"
server_key_iv: "2c8976e849e15f44d05e522ad5804e63"
server_key_version: "encryptionkey@0"
I20260928 21:59:21.994071 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:22.013180 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:22.015165 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:22.018179 9267 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.125:33015
I20260928 21:59:22.018200 9436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.125:33015 every 8 connection(s)
I20260928 21:59:22.018548 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
I20260928 21:59:22.019515 9437 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:59:22.020062 9437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Bootstrap starting.
I20260928 21:59:22.023077 9437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:59:22.023300 9437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: Bootstrap complete.
I20260928 21:59:22.023502 9437 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } }
I20260928 21:59:22.023614 9437 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0c923da36c948d39913ba44657480cd, State: Initialized, Role: LEARNER
I20260928 21:59:22.023655 9437 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } }
I20260928 21:59:22.023887 9437 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:22.023873 9440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } }
I20260928 21:59:22.024047 9440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: LEARNER
I20260928 21:59:22.025401 9451 catalog_manager.cc:1259] Loaded cluster ID: b3f89f07593f487d8dd67a400cac3541
I20260928 21:59:22.025454 9451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: loading cluster ID for follower catalog manager: success
I20260928 21:59:22.026628 9451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: acquiring CA information for follower catalog manager: success
I20260928 21:59:22.026912 9451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:59:22.071841 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.064931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.2.165.125:33015)
0928 21:59:22.065073 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.065075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.065090 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:59:22.065113 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.065114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.065123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.065201 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.065793 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.066468 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.067230 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.067459 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.067541 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.068899 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.068901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.068901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.068917 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.070588 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.070951 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.070952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.070952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.071021 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.071220 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.071221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.071221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.071364 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.071483 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.071640 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.071725 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.2.165.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.124.keytab.
Entry for principal oryx/127.2.165.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.124.keytab.
I20260928 21:59:22.099606 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:43475
--webserver_interface=127.2.165.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.124.keytab
--principal=oryx/127.2.165.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:22.185134 9455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:22.185357 9455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:22.185380 9455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:22.186635 9455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:22.186681 9455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:22.186695 9455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:22.186708 9455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:22.186748 9455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:22.188265 9455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.124.keytab
--principal=oryx/127.2.165.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.165.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:22.188524 9455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:22.188862 9455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:22.193783 9463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.193783 9460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.193786 9461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:22.194095 9455 server_base.cc:1034] running on GCE node
I20260928 21:59:22.194367 9455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:22.194669 9455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:22.195839 9455 hybrid_clock.cc:648] HybridClock initialized: now 1790632762195828 us; error 43 us; skew 500 ppm
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:22.198482 9455 init.cc:377] Logged in from keytab as oryx/127.2.165.124@KRBTEST.COM (short username oryx)
I20260928 21:59:22.198946 9455 webserver.cc:460] Webserver started at http://127.2.165.124:38273/ using document root <none> and password file <none>
I20260928 21:59:22.199175 9455 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:22.199242 9455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:22.199389 9455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:22.200488 9455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/instance:
uuid: "98c59779fe4f47c09127ff1dad13c909"
format_stamp: "Formatted at 2026-09-28 21:59:22 on dist-test-slave-3t36"
server_key: "9d2045fb8ce45cc3fd370e4472c1f2b2"
server_key_iv: "ee2c6da539b10e9c6234a7f0a8942a88"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.200894 9455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal/instance:
uuid: "98c59779fe4f47c09127ff1dad13c909"
format_stamp: "Formatted at 2026-09-28 21:59:22 on dist-test-slave-3t36"
server_key: "9d2045fb8ce45cc3fd370e4472c1f2b2"
server_key_iv: "ee2c6da539b10e9c6234a7f0a8942a88"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.202287 9455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:22.203111 9470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.203290 9455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:22.203429 9455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
uuid: "98c59779fe4f47c09127ff1dad13c909"
format_stamp: "Formatted at 2026-09-28 21:59:22 on dist-test-slave-3t36"
server_key: "9d2045fb8ce45cc3fd370e4472c1f2b2"
server_key_iv: "ee2c6da539b10e9c6234a7f0a8942a88"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.203508 9455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:22.218142 9455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:22.220780 9455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:22.220966 9455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:22.225968 9455 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.124:43475
I20260928 21:59:22.225999 9522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.124:43475 every 8 connection(s)
I20260928 21:59:22.226394 9455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
I20260928 21:59:22.227470 9523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:22.229125 9523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.124@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:22.235587 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 9455
I20260928 21:59:22.239193 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.229998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.2.165.126:34319)
0928 21:59:22.230081 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.230082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.230279 (+ 197us) server_negotiation.cc:407] Connection header received
0928 21:59:22.230780 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.230781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.230824 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.230889 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.231536 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.232053 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.233153 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.233341 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.233399 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.235810 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.235811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.235811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.235818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.237710 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.238121 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.238122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.238123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.238185 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.238532 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.238533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.238533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.238650 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.238711 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.239058 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.239127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:59:22.239456 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.229917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53158)
0928 21:59:22.230150 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.230187 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.230671 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.230970 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.230974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.231226 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.231434 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.231438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.232139 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.232140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.233077 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.233079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.233462 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.233463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.233591 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.234086 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.234095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.235710 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.237799 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.237805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.237807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.238043 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.238251 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.238252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.238252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.238368 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.238724 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.238731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.238897 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.239014 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.239184 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260928 21:59:22.239998 9523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.165.124@KRBTEST.COM
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.124@KRBTEST.COM for oryx/127.2.165.125@KRBTEST.COM
I20260928 21:59:22.245478 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.236209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.2.165.124:43475)
0928 21:59:22.236472 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.236477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.236501 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:59:22.236524 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.236526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.236550 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.236647 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.237346 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.238334 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.239327 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.239774 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.239850 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.242616 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.242623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.242623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.242630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.244241 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.244622 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.244626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.244626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.244727 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.244914 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.244915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.244915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.245067 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.245221 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.245291 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.245362 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":155,"threads_started":1}
I20260928 21:59:22.247287 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.240492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.0.0.1:42864)
0928 21:59:22.240552 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.240564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.240607 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.240920 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.240921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.241084 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.241147 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.241149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.241886 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.241887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.242497 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.242499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.242872 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.242874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.242953 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.243324 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.243328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.244547 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.246371 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.246372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.246372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.246554 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.246747 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.246748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.246748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.246799 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.247051 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.247052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.247087 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.247152 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.247224 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:59:22.247300 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.240753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.2.165.125:33015)
0928 21:59:22.240794 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.240797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.240807 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:22.240836 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.240837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.240847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.240891 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.241229 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.241791 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.242584 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.242798 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.242876 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.244633 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.244634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.244635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.244642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.246306 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.246641 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.246642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.246642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.246692 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.246865 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.246865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.246866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.246970 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.247060 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.247175 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.247241 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:59:22.247660 9523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.124@KRBTEST.COM for oryx/127.2.165.124@KRBTEST.COM
I20260928 21:59:22.253607 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.247909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.0.0.1:57354)
0928 21:59:22.247965 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.247977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.248021 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.248136 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.248137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.248265 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.248290 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.248291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.249070 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.249071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.249538 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.249539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.249921 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.249921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.249968 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.250375 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.250380 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.251478 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.252784 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.252785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.252785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.252945 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.253116 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.253117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.253117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.253165 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.253357 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.253358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.253389 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.253478 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.253542 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:59:22.253679 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.247964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.2.165.124:43475)
0928 21:59:22.247993 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.247995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.248006 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:22.248080 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.248081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.248089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.248136 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.248358 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.248977 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.249631 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.249835 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.249918 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.251557 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.251559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.251559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.251565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.252651 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.253016 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.253017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.253017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.253063 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.253216 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.253216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.253216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.253307 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.253353 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.253544 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.253603 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:59:22.255167 9523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Bootstrap starting.
I20260928 21:59:22.256006 9523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:22.256237 9523 log.cc:824] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:22.256986 9523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: No bootstrap required, opened a new log
I20260928 21:59:22.258635 9523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } }
I20260928 21:59:22.258764 9523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:22.258787 9523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c59779fe4f47c09127ff1dad13c909, State: Initialized, Role: FOLLOWER
I20260928 21:59:22.258876 9523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } }
I20260928 21:59:22.259120 9528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } } }
I20260928 21:59:22.259184 9523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:22.259204 9528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:59:22.260906 9539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:59:22.261005 9539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:59:22.263701 9455 master_runner.cc:428] Detected that this master 98c59779fe4f47c09127ff1dad13c909 is joining an existing cluster
I20260928 21:59:22.263764 9455 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.165.124:43475
I20260928 21:59:22.272303 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.265068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.2.165.124:43475)
0928 21:59:22.265102 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.265104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.265117 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:59:22.265187 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.265188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.265197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.265254 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.265666 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.266411 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.267449 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.267720 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.267777 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.269427 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.269428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.269429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.269435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.271119 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.271522 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.271524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.271524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.271580 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.271826 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.271827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.271827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.271946 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.272015 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.272164 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.272236 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:59:22.272488 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.264883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.0.0.1:57364)
0928 21:59:22.265068 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.265080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.265130 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.265295 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.265296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.265513 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.265584 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.265586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.266499 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.266500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.267346 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.267348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.267835 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.267836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.267890 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.268863 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.268869 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.269324 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.271197 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.271198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.271198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.271431 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.271655 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.271656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.271656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.271732 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.272018 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.272019 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.272069 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.272143 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.272303 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":54,"thread_start_us":100,"threads_started":1}
I20260928 21:59:22.293704 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.264941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.2.165.126:34319)
0928 21:59:22.264997 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.265000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.265286 (+ 286us) server_negotiation.cc:407] Connection header received
0928 21:59:22.265392 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.265393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.265406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.265438 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.266129 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.266866 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.267935 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.268232 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.268268 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.290454 (+ 22186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.290457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.290457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.290467 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.292227 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.292733 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.292734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.292734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.292802 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.293027 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.293029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.293030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.293217 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.293289 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.293518 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.293608 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:59:22.293713 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.264895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53166)
0928 21:59:22.265235 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.265244 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.265310 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.265507 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.265508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.265997 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.266049 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.266050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.266979 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.266980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.267822 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.267824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.268345 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.268345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.268429 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.289477 (+ 21048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.289484 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.290300 (+ 816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.292374 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.292376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.292376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.292621 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.292881 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.292882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.292882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.292948 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.293294 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.293295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.293349 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.293464 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.293590 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260928 21:59:22.293733 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.265152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.2.165.125:33015)
0928 21:59:22.265193 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.265195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.265439 (+ 244us) server_negotiation.cc:407] Connection header received
0928 21:59:22.265535 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.265535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.265546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.265576 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.266019 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.266785 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.267923 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.268218 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.268262 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.290787 (+ 22525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.290789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.290790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.290798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.292471 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.292891 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.292892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.292892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.292961 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.293177 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.293178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.293178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.293321 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.293384 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.293567 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.293645 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:59:22.294384 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.265252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.0.0.1:42874)
0928 21:59:22.265370 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.265380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.265463 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.265637 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.265638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.265856 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.265923 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.265924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.266882 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.266882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.267828 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.267829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.268343 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.268344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.268429 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.289570 (+ 21141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.289574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.290296 (+ 722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.292579 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.292580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.292580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.292799 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.293042 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.293042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.293043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.293090 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.293376 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.293377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.293422 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.294147 (+ 725us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.294266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":65,"thread_start_us":63,"threads_started":1}
I20260928 21:59:22.296816 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8865
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632761, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.126@KRBTEST.COM for oryx/127.2.165.124@KRBTEST.COM
I20260928 21:59:22.303063 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:40909
--local_ip_for_outbound_sockets=127.2.165.65
--tserver_master_addrs=127.2.165.126:34319
--webserver_port=39183
--webserver_interface=127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab
--principal=oryx/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:59:22.305846 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.296480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.0.0.1:57372)
0928 21:59:22.296579 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.296591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.296642 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.296837 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.296838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.297019 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.297073 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.297076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.298371 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.298372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.299210 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.299213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.299706 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.299707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.299815 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.300338 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.300344 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.302037 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.304544 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.304545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.304546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.304840 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.305082 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.305083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.305083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.305156 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.305455 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.305456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.305515 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.305584 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.305703 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 21:59:22.305868 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.296590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.2.165.124:43475)
0928 21:59:22.296627 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.296629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.296638 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:22.296738 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.296739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.296749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.297512 (+ 763us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.297557 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.298254 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.299312 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.299610 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.299696 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.302140 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.302142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.302142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.302150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.304315 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.304934 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.304936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.304936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.305002 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.305232 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.305233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.305233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.305384 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.305449 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.305654 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.305774 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:59:22.306701 8803 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.165.124:43475: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "98c59779fe4f47c09127ff1dad13c909"
member_type: NON_VOTER
last_known_addr {
host: "127.2.165.124"
port: 43475
}
attrs {
promote: true
}
}
dest_uuid: "4cbf333743754a5f80314e9e2eae266f"
cas_config_opid_index: 5
I20260928 21:59:22.306983 8803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } }
I20260928 21:59:22.307569 9363 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.309469 9363 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.165.124:43475
I20260928 21:59:22.309562 9363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.309652 9363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:22.309619 9497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:22.310053 9455 master.cc:537] Master@127.2.165.124:43475 shutting down...
I20260928 21:59:22.310608 9497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Refusing update from remote peer 4cbf333743754a5f80314e9e2eae266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:59:22.310953 9362 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:59:22.312045 8787 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f -> Peer 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124:43475): Couldn't send request to peer 98c59779fe4f47c09127ff1dad13c909. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:59:22.313251 9455 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:22.313342 9455 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:22.313392 9455 tablet_replica.cc:330] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: stopping tablet replica
I20260928 21:59:22.315263 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.308987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.2.165.125:33015)
0928 21:59:22.310014 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.310016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.310152 (+ 136us) server_negotiation.cc:407] Connection header received
0928 21:59:22.310335 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.310336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.310361 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.310418 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:22.310607 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.311381 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.313256 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.314196 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.314277 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.314425 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.315119 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.315167 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995}
I20260928 21:59:22.315565 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.308864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.0.0.1:42884)
0928 21:59:22.310047 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.310059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.310169 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.310480 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.310481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.310483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:22.310529 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.310531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.311602 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.311604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.312980 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.312983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.314481 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.314483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.314563 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.314592 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.314701 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.314756 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166}
I20260928 21:59:22.316108 9411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 LEARNER]: Refusing update from remote peer 4cbf333743754a5f80314e9e2eae266f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:59:22.316357 9362 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 21:59:22.317265 9411 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.318827 9440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbf333743754a5f80314e9e2eae266f. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.318930 9440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: LEARNER
I20260928 21:59:22.318969 9440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: NON_VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: true } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.319010 9440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: LEARNER
I20260928 21:59:22.319180 9362 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: attempting to promote NON_VOTER e0c923da36c948d39913ba44657480cd to VOTER
I20260928 21:59:22.319403 9362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } }
I20260928 21:59:22.319947 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.319827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.0.0.1:57376)
0928 21:59:22.319867 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.319918 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:43475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:59:22.320293 9411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 LEARNER]: Refusing update from remote peer 4cbf333743754a5f80314e9e2eae266f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 21:59:22.320397 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.320298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.0.0.1:57378)
0928 21:59:22.320337 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.320359 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:43475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 21:59:22.320461 8787 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f -> Peer 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124:43475): Couldn't send request to peer 98c59779fe4f47c09127ff1dad13c909. Status: Network error: Client connection negotiation failed: client connection to 127.2.165.124:43475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:59:22.320608 9549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 21:59:22.321314 9363 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e0c923da36c948d39913ba44657480cd (127.2.165.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.321491 9411 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e0c923da36c948d39913ba44657480cd (127.2.165.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.322312 9363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.322382 9363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:22.322429 9363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.322477 9363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:22.322530 9551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.322614 9551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:59:22.326151 9455 master.cc:552] Master@127.2.165.124:43475 shutdown complete.
I20260928 21:59:22.326197 9455 master_runner.cc:305] Clearing existing system tablet
I20260928 21:59:22.326342 9455 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:22.326392 9455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:22.327056 9455 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:22.327629 9554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.327812 9455 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:22.327874 9455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:22.327899 9455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
uuid: "98c59779fe4f47c09127ff1dad13c909"
format_stamp: "Formatted at 2026-09-28 21:59:22 on dist-test-slave-3t36"
server_key: "9d2045fb8ce45cc3fd370e4472c1f2b2"
server_key_iv: "ee2c6da539b10e9c6234a7f0a8942a88"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.328660 9455 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:22.330277 9455 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:59:22.330348 9455 log.cc:1196] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 21:59:22.330556 9455 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Deleting consensus metadata
I20260928 21:59:22.330677 9455 master_runner.cc:315] Copying system tablet from 127.2.165.126:34319
I20260928 21:59:22.331265 9455 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: tablet copy: Beginning tablet copy session from remote peer at address 127.2.165.126:34319
I20260928 21:59:22.338956 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.331612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53172)
0928 21:59:22.331759 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.331773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.331842 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.332005 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.332006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.332218 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.332270 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.332272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.333239 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.333241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.334198 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.334201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.334674 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.334675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.334753 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.335314 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.335319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.335723 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.337779 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.337780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.337780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.337998 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.338232 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.338233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.338233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.338314 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.338610 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.338611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.338661 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.338740 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.338852 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260928 21:59:22.338959 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.331791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.2.165.126:34319)
0928 21:59:22.331838 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.331840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.331851 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:22.331915 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.331916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.331926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.331990 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.332353 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.333135 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.334300 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.334584 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.334655 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.335823 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.335825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.335825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.335833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.337691 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.338091 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.338092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.338092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.338158 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.338386 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.338387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.338388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.338530 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.338599 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.338781 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.338863 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:59:22.339660 8823 tablet_copy_service.cc:140] P 4cbf333743754a5f80314e9e2eae266f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 98c59779fe4f47c09127ff1dad13c909 ({username='oryx', principal='oryx/127.2.165.124@KRBTEST.COM'} at 127.0.0.1:53172)
I20260928 21:59:22.339730 8823 tablet_copy_service.cc:161] P 4cbf333743754a5f80314e9e2eae266f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 98c59779fe4f47c09127ff1dad13c909 at {username='oryx', principal='oryx/127.2.165.124@KRBTEST.COM'} at 127.0.0.1:53172: session id = 98c59779fe4f47c09127ff1dad13c909-00000000000000000000000000000000
I20260928 21:59:22.340330 8823 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:59:22.340880 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:59:22.342283 9455 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: tablet copy: Starting download of 0 data blocks...
I20260928 21:59:22.342481 9455 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: tablet copy: Starting download of 1 WAL segments...
I20260928 21:59:22.343411 9455 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:59:22.344348 8823 tablet_copy_service.cc:342] P 4cbf333743754a5f80314e9e2eae266f: Request end of tablet copy session 98c59779fe4f47c09127ff1dad13c909-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.165.124@KRBTEST.COM'} at 127.0.0.1:53172
I20260928 21:59:22.344417 8823 tablet_copy_service.cc:433] P 4cbf333743754a5f80314e9e2eae266f: ending tablet copy session 98c59779fe4f47c09127ff1dad13c909-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 98c59779fe4f47c09127ff1dad13c909
I20260928 21:59:22.348585 9455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:22.350087 9561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.353737 9562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:22.353991 9455 server_base.cc:1034] running on GCE node
W20260928 21:59:22.356858 9564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:22.357076 9455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:22.357236 9455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:22.358366 9455 hybrid_clock.cc:648] HybridClock initialized: now 1790632762358343 us; error 46 us; skew 500 ppm
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:22.360023 9455 init.cc:377] Logged in from keytab as oryx/127.2.165.124@KRBTEST.COM (short username oryx)
I20260928 21:59:22.360337 9455 webserver.cc:460] Webserver started at http://127.2.165.124:34739/ using document root <none> and password file <none>
I20260928 21:59:22.360466 9455 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:22.360507 9455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:22.361518 9455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:22.362147 9571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.362257 9455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:22.362291 9455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
uuid: "98c59779fe4f47c09127ff1dad13c909"
format_stamp: "Formatted at 2026-09-28 21:59:22 on dist-test-slave-3t36"
server_key: "9d2045fb8ce45cc3fd370e4472c1f2b2"
server_key_iv: "ee2c6da539b10e9c6234a7f0a8942a88"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.362352 9455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:22.376557 9455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:22.378487 9455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:22.381461 9455 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.124:43475
I20260928 21:59:22.381841 9455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
I20260928 21:59:22.384927 9623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.124:43475 every 8 connection(s)
I20260928 21:59:22.385727 9624 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:59:22.386312 9624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Bootstrap starting.
I20260928 21:59:22.389331 9624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:59:22.389559 9624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Bootstrap complete.
I20260928 21:59:22.389765 9624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } }
I20260928 21:59:22.389873 9624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98c59779fe4f47c09127ff1dad13c909, State: Initialized, Role: LEARNER
I20260928 21:59:22.389910 9624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } }
I20260928 21:59:22.390130 9624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:22.390125 9627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.390407 9627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:59:22.391808 9638 catalog_manager.cc:1259] Loaded cluster ID: b3f89f07593f487d8dd67a400cac3541
I20260928 21:59:22.392220 9638 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: loading cluster ID for follower catalog manager: success
I20260928 21:59:22.393534 9638 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: acquiring CA information for follower catalog manager: success
I20260928 21:59:22.394037 9638 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:59:22.410782 9548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:22.411046 9548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:22.411090 9548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:22.412334 9548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:22.412406 9548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:22.412465 9548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:22.414044 9548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab
--principal=oryx/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=39183
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:22.414323 9548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:22.414573 9548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:22.420538 9644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.420538 9646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.420542 9643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:22.421169 9548 server_base.cc:1034] running on GCE node
I20260928 21:59:22.421490 9548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:22.421768 9548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:22.422960 9548 hybrid_clock.cc:648] HybridClock initialized: now 1790632762422936 us; error 40 us; skew 500 ppm
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:22.426853 9548 init.cc:377] Logged in from keytab as oryx/127.2.165.65@KRBTEST.COM (short username oryx)
I20260928 21:59:22.427309 9548 webserver.cc:460] Webserver started at http://127.2.165.65:39183/ using document root <none> and password file <none>
I20260928 21:59:22.427554 9548 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:22.427647 9548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:22.429143 9548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:22.429826 9653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.430001 9548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:22.430117 9548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "fc9c2ec29eb540fbae683aa694dd6c6b"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "b5642463da4c7331c2c175c7a80e99b8"
server_key_iv: "e59803b38a6b58559ffa29a75b4dba78"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.430523 9548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:22.437268 9548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:22.439348 9548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:22.439519 9548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:22.439761 9548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:22.440068 9548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:22.440124 9548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.440167 9548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:22.440193 9548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.446501 9548 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:40909
I20260928 21:59:22.446622 9766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:40909 every 8 connection(s)
I20260928 21:59:22.446949 9548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:22.447154 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 9548
I20260928 21:59:22.447316 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8999
I20260928 21:59:22.452786 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:38935
--local_ip_for_outbound_sockets=127.2.165.66
--tserver_master_addrs=127.2.165.126:34319
--webserver_port=41211
--webserver_interface=127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab
--principal=oryx/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.124@KRBTEST.COM
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.125@KRBTEST.COM
I20260928 21:59:22.459600 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.448218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.2.165.65:60435)
0928 21:59:22.448400 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.448451 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.449164 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.449523 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.449527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.450133 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.450339 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.450344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.451228 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.451229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.452066 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.452068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.452494 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.452495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.452675 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.453326 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.453338 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.455260 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.457512 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.457521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.457524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.457981 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.458311 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.458315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.458315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.458469 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.458805 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.458811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.458993 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.459117 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.459305 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260928 21:59:22.460178 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.449162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:60435 (local address 127.2.165.124:43475)
0928 21:59:22.449295 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.449297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.449307 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:22.449334 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.449334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.449346 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.449414 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.450448 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.451107 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.452172 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.452396 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.452480 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.455367 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.455369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.455370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.455378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.457410 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.458086 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.458088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.458088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.458219 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.458554 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.458555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.458555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.458713 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.458778 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.459957 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.460059 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260928 21:59:22.461016 9767 heartbeater.cc:347] Connected to a master server at 127.2.165.124:43475
I20260928 21:59:22.461130 9767 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.461345 9767 heartbeater.cc:511] Master 127.2.165.124:43475 requested a full tablet report, sending...
I20260928 21:59:22.461970 9584 ts_manager.cc:194] Registered new tserver with Master: fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
I20260928 21:59:22.462325 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.448847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:51027 (local address 127.2.165.126:34319)
0928 21:59:22.448895 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.448897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.452598 (+ 3701us) server_negotiation.cc:407] Connection header received
0928 21:59:22.452652 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.452653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.452666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.452741 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.453527 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.454276 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.455323 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.455602 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.455647 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.458493 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.458494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.458494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.458503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.460604 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.461075 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.461076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.461077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.461137 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.461736 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.461738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.461738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.461869 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.461922 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.462145 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.462232 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:59:22.462447 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.448859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:55503 (local address 127.2.165.125:33015)
0928 21:59:22.448903 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.448906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.452760 (+ 3854us) server_negotiation.cc:407] Connection header received
0928 21:59:22.452860 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.452862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.452874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.452907 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.453421 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.454195 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.455156 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.455466 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.455543 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.458605 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.458606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.458607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.458615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.460329 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.460798 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.460799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.460799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.460863 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.461707 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.461708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.461709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.461884 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.461944 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.462221 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.462313 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:59:22.462853 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.448506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:51027)
0928 21:59:22.452510 (+ 4004us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.452523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.452590 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.452788 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.452790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.453349 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.453413 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.453415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.454378 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.454379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.455214 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.455216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.455727 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.455728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.455788 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.456136 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.456142 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.458395 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.460737 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.460738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.460738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.460972 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.461216 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.461217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.461217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.461654 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.461958 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.461958 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.462012 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.462692 (+ 680us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.462784 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3980,"thread_start_us":62,"threads_started":1}
I20260928 21:59:22.463039 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.448650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.2.165.65:55503)
0928 21:59:22.452705 (+ 4055us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.452714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.452786 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.452973 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.452974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.453241 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.453328 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.453330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.454293 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.454293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.455071 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.455074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.455606 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.455606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.456162 (+ 556us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.456782 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.456787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.458498 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.460434 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.460435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.460435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.460702 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.460943 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.460944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.460944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.461609 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.462039 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.462040 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.462090 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.462671 (+ 581us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.462862 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4016,"mutex_wait_us":58,"thread_start_us":70,"threads_started":1}
I20260928 21:59:22.463459 9768 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34319
I20260928 21:59:22.463511 9768 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.463590 9768 heartbeater.cc:511] Master 127.2.165.126:34319 requested a full tablet report, sending...
I20260928 21:59:22.463747 9769 heartbeater.cc:347] Connected to a master server at 127.2.165.125:33015
I20260928 21:59:22.463786 9769 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.463848 9769 heartbeater.cc:511] Master 127.2.165.125:33015 requested a full tablet report, sending...
I20260928 21:59:22.463871 8803 ts_manager.cc:194] Re-registered known tserver with Master: fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
I20260928 21:59:22.464263 8803 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:51027
I20260928 21:59:22.464283 9401 ts_manager.cc:194] Registered new tserver with Master: fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
W20260928 21:59:22.542025 9775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:22.542243 9775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:22.542268 9775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:22.543428 9775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:22.543468 9775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:22.543495 9775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:22.545037 9775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab
--principal=oryx/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:38935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=41211
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:22.545326 9775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:22.545632 9775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:22.551609 9780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.551648 9781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.551636 9783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:22.551688 9775 server_base.cc:1034] running on GCE node
I20260928 21:59:22.552079 9775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:22.552337 9775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:22.553526 9775 hybrid_clock.cc:648] HybridClock initialized: now 1790632762553512 us; error 31 us; skew 500 ppm
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:22.556252 9775 init.cc:377] Logged in from keytab as oryx/127.2.165.66@KRBTEST.COM (short username oryx)
I20260928 21:59:22.556838 9775 webserver.cc:460] Webserver started at http://127.2.165.66:41211/ using document root <none> and password file <none>
I20260928 21:59:22.557096 9775 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:22.557183 9775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:22.558809 9775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:22.559706 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.560089 9775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:22.560209 9775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "0bf075309fb547d08fce67d52f331086"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "a5316d2c38dfd948e9bb70588d8fec43"
server_key_iv: "0c6844fcedce9a60d68a5969b0c359bf"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.560618 9775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:22.587508 9775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:22.590260 9775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:22.590438 9775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:22.590687 9775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:22.591048 9775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:22.591084 9775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.591135 9775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:22.591161 9775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.598135 9775 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:38935
I20260928 21:59:22.598196 9903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:38935 every 8 connection(s)
I20260928 21:59:22.598609 9775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:22.601222 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 9775
I20260928 21:59:22.601378 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 9133
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.125@KRBTEST.COM
I20260928 21:59:22.607550 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:42575
--local_ip_for_outbound_sockets=127.2.165.67
--tserver_master_addrs=127.2.165.126:34319
--webserver_port=45243
--webserver_interface=127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab
--principal=oryx/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.124@KRBTEST.COM
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:22.612278 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.599486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:45025 (local address 127.2.165.125:33015)
0928 21:59:22.599529 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.599531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.600585 (+ 1054us) server_negotiation.cc:407] Connection header received
0928 21:59:22.601064 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.601065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.601078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.601111 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.602405 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.603117 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.604733 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.605089 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.605168 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.608893 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.608895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.608896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.608905 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.610578 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611088 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.611090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.611090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.611148 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611422 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.611423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.611423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.611586 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.611631 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.612039 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.612135 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:59:22.612422 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.599641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:59265 (local address 127.2.165.124:43475)
0928 21:59:22.599718 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.599720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.599944 (+ 224us) server_negotiation.cc:407] Connection header received
0928 21:59:22.600651 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.600652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.600666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.600739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.602303 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.603119 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.605557 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.606219 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.606264 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.608783 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.608785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.608785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.608794 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.611191 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611582 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.611583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.611583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.611636 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611835 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.611836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.611836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.611973 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.612028 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.612225 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.612322 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:59:22.612545 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.599774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:47053 (local address 127.2.165.126:34319)
0928 21:59:22.599821 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.599823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.599938 (+ 115us) server_negotiation.cc:407] Connection header received
0928 21:59:22.600482 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.600483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.600496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.600598 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.605488 (+ 4890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.606183 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.607155 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.607459 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.607504 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.609678 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.609680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.609680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.609688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.611374 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611747 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.611748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.611749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.611801 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611988 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.611989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.611989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.612131 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.612173 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.612358 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.612429 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:59:22.613020 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.599692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:47053)
0928 21:59:22.599872 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.599882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.600355 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.600761 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.600767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.601310 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.605379 (+ 4069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.605381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.606285 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.606286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.607053 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.607055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.607587 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.607588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.607663 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.608117 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.608122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.609568 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.611454 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.611456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.611665 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.611873 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.611873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.611920 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.612217 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.612218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.612250 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.612653 (+ 403us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.612837 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":67,"thread_start_us":63,"threads_started":1}
I20260928 21:59:22.613026 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.599405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.2.165.66:59265)
0928 21:59:22.599806 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.599860 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.600382 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.600769 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.600771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.602038 (+ 1267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.602188 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.602194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.604659 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.604659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.605455 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.605457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.606341 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.606342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.606416 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.606856 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.606861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.608625 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.611286 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.611288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.611502 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.611709 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.611711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.611762 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.612076 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.612077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.612114 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.612550 (+ 436us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.612777 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260928 21:59:22.613480 9904 heartbeater.cc:347] Connected to a master server at 127.2.165.124:43475
I20260928 21:59:22.613488 9905 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34319
I20260928 21:59:22.613603 9904 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.613603 9905 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.613829 9905 heartbeater.cc:511] Master 127.2.165.126:34319 requested a full tablet report, sending...
I20260928 21:59:22.613982 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.599398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.2.165.66:45025)
0928 21:59:22.600527 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.600536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.600928 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.601173 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.601173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.601375 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.602309 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.602312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.603244 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.603245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.604558 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.604561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.605241 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.605242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.605412 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.606066 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.606077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.608789 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.610663 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.610671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.610674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.611001 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.611215 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.611216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.611216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.611353 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.611680 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.611686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.611871 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.613015 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.613630 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260928 21:59:22.613829 9904 heartbeater.cc:511] Master 127.2.165.124:43475 requested a full tablet report, sending...
I20260928 21:59:22.614233 8803 ts_manager.cc:194] Re-registered known tserver with Master: 0bf075309fb547d08fce67d52f331086 (127.2.165.66:38935)
I20260928 21:59:22.614284 9584 ts_manager.cc:194] Registered new tserver with Master: 0bf075309fb547d08fce67d52f331086 (127.2.165.66:38935)
I20260928 21:59:22.614791 8803 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:47053
I20260928 21:59:22.615212 9906 heartbeater.cc:347] Connected to a master server at 127.2.165.125:33015
I20260928 21:59:22.615262 9906 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.615392 9906 heartbeater.cc:511] Master 127.2.165.125:33015 requested a full tablet report, sending...
I20260928 21:59:22.615821 9401 ts_manager.cc:194] Registered new tserver with Master: 0bf075309fb547d08fce67d52f331086 (127.2.165.66:38935)
W20260928 21:59:22.693284 9911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:22.693511 9911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:22.693535 9911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:22.694828 9911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:22.694877 9911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:22.694904 9911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:22.696534 9911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab
--principal=oryx/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:42575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=45243
--tserver_master_addrs=127.2.165.126:34319,127.2.165.125:33015,127.2.165.124:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:22.696853 9911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:22.697117 9911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:22.702522 9919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.702522 9917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:22.702529 9916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:22.703082 9911 server_base.cc:1034] running on GCE node
I20260928 21:59:22.703320 9911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:22.703613 9911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:22.704756 9911 hybrid_clock.cc:648] HybridClock initialized: now 1790632762704738 us; error 51 us; skew 500 ppm
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:22.707554 9911 init.cc:377] Logged in from keytab as oryx/127.2.165.67@KRBTEST.COM (short username oryx)
I20260928 21:59:22.708076 9911 webserver.cc:460] Webserver started at http://127.2.165.67:45243/ using document root <none> and password file <none>
I20260928 21:59:22.708323 9911 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:22.708410 9911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:22.709944 9911 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:22.710721 9926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.710906 9911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:22.711009 9911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6"
format_stamp: "Formatted at 2026-09-28 21:59:21 on dist-test-slave-3t36"
server_key: "8ecd339997536dd178c939b2c040287c"
server_key_iv: "c34d2c1e004896d04394fa4053eeb3b0"
server_key_version: "encryptionkey@0"
I20260928 21:59:22.711410 9911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:22.732020 9911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:22.734710 9911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:22.734889 9911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:22.735137 9911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:22.735466 9911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:22.735528 9911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.735600 9911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:22.735630 9911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:22.741998 9911 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:42575
I20260928 21:59:22.742041 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:42575 every 8 connection(s)
I20260928 21:59:22.742486 9911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:22.743913 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 9911
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.124@KRBTEST.COM
Sep 28 21:59:22 dist-test-slave-3t36 krb5kdc[8758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632762, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.125@KRBTEST.COM
I20260928 21:59:22.756129 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.745123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.2.165.124:43475)
0928 21:59:22.745450 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.745452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.745459 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:59:22.745534 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.745535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.745546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.746308 (+ 762us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.746524 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.747291 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.748856 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.749153 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.749217 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.751886 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.751888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.751888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.751897 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.754697 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.755116 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.755117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.755117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.755179 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.755391 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.755392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.755392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.755544 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.755886 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.755926 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.756011 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":58,"threads_started":1}
I20260928 21:59:22.756925 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.743337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:32851 (local address 127.2.165.124:43475)
0928 21:59:22.743377 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.743379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.743655 (+ 276us) server_negotiation.cc:407] Connection header received
0928 21:59:22.744333 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.744335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.744349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.744424 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.746048 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.746848 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.748461 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.748765 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.748818 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.753435 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.753437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.753437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.753444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.755278 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.755797 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.755799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.755799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.755851 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756156 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.756157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.756157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.756276 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.756322 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.756726 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.756811 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:59:22.757540 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.743757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:34679 (local address 127.2.165.125:33015)
0928 21:59:22.743811 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.743813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.745169 (+ 1356us) server_negotiation.cc:407] Connection header received
0928 21:59:22.745304 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.745312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.745324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.745357 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.748483 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.749173 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.750199 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.750515 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.750560 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.753454 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.753455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.753455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.753462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.756116 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756653 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.756655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.756656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.756741 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756943 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.756944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.756944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.757079 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.757125 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.757349 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.757444 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:59:22.757558 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.743312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.2.165.67:32851)
0928 21:59:22.743503 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.743554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.744215 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.744492 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.744496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.745522 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.745910 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.745917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.746993 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.746993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.748358 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.748360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.748903 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.748903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.749057 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.750300 (+ 1243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.750304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.753334 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.755372 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.755383 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.755387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.755714 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.755923 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.755924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.755925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.756076 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.756360 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.756365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.756546 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.757061 (+ 515us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.757303 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":3,"spinlock_wait_cycles":8448,"thread_start_us":103,"threads_started":1}
I20260928 21:59:22.757622 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.743291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:37467 (local address 127.2.165.126:34319)
0928 21:59:22.743345 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.743348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.743654 (+ 306us) server_negotiation.cc:407] Connection header received
0928 21:59:22.744490 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.744492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.744504 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.744561 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.746523 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.747234 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.748431 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.748743 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.748806 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.753159 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.753161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.753161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.753169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.755545 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756608 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.756609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.756609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.756689 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756904 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.756905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.756906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.757055 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.757102 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.757442 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.757523 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:59:22.758018 10040 heartbeater.cc:347] Connected to a master server at 127.2.165.124:43475
I20260928 21:59:22.757558 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.743606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.2.165.67:34679)
0928 21:59:22.743720 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.745109 (+ 1389us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.745218 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.745427 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.745428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.747430 (+ 2002us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.747498 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.747500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.749271 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.749272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.750097 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.750100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.750638 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.750639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.750714 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.751116 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.751121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.753365 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.756271 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.756272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.756563 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.756819 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.756820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.756875 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.757172 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.757173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.757223 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.757312 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.757392 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
I20260928 21:59:22.758142 10040 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.758006 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.743312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:37467)
0928 21:59:22.743503 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.743554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.744214 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.744615 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.744616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.746360 (+ 1744us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:22.746418 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.746420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.747596 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.747597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.748364 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.748365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.749272 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.749273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.749345 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.749833 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:22.749845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:22.753039 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:22.756004 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.756005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.756512 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.756769 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:22.756770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:22.756771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:22.756831 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:22.757276 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:22.757277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:22.757322 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.757705 (+ 383us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.757805 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260928 21:59:22.758585 10040 heartbeater.cc:511] Master 127.2.165.124:43475 requested a full tablet report, sending...
I20260928 21:59:22.759018 9584 ts_manager.cc:194] Registered new tserver with Master: db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67:42575)
I20260928 21:59:22.759078 10041 heartbeater.cc:347] Connected to a master server at 127.2.165.126:34319
I20260928 21:59:22.759130 10041 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.759205 10041 heartbeater.cc:511] Master 127.2.165.126:34319 requested a full tablet report, sending...
I20260928 21:59:22.759470 10042 heartbeater.cc:347] Connected to a master server at 127.2.165.125:33015
I20260928 21:59:22.759517 10042 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:22.759522 8803 ts_manager.cc:194] Re-registered known tserver with Master: db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67:42575)
I20260928 21:59:22.759579 10042 heartbeater.cc:511] Master 127.2.165.125:33015 requested a full tablet report, sending...
I20260928 21:59:22.759605 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.745599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.2.165.125:33015)
0928 21:59:22.745847 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.745849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.745979 (+ 130us) server_negotiation.cc:407] Connection header received
0928 21:59:22.746084 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.746085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.746098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.746152 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.747031 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.747756 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.750861 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.751166 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.751234 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.753000 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.753002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.753002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.753010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.756852 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.758544 (+ 1692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.758545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.758546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.758612 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.758824 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.758825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.758825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.758944 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.759035 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.759439 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.759522 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":66,"threads_started":1}
I20260928 21:59:22.760010 8803 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:37467
I20260928 21:59:22.760195 9401 ts_manager.cc:194] Registered new tserver with Master: db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67:42575)
I20260928 21:59:22.761960 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.745098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.2.165.126:34319)
0928 21:59:22.745810 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.745812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.745821 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:22.745847 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.745848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.745861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.745953 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.748526 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.751629 (+ 3103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.752716 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.753009 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.753075 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.758138 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.758139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.758140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.758148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.760819 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.761159 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.761160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.761160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.761220 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.761419 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.761420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.761421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.761533 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.761634 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.761758 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.761845 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":81,"threads_started":1}
I20260928 21:59:22.763598 8803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:22.765256 8803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:22.773789 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.769826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.2.165.65:40909)
0928 21:59:22.770065 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.770069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.770255 (+ 186us) server_negotiation.cc:407] Connection header received
0928 21:59:22.770329 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.770331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.770346 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.770428 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:22.770599 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.771639 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.772914 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.773421 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.773512 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.773593 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.773663 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.773702 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
I20260928 21:59:22.773887 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.769428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:42575 (local address 127.0.0.1:43328)
0928 21:59:22.769604 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.769618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.769661 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.770195 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.770196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.770198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:22.770252 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.770254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.771667 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.771668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.772612 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.772614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.773621 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.773622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.773682 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.773685 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.773715 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.773763 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":56}
I20260928 21:59:22.774068 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.769752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.2.165.67:42575)
0928 21:59:22.770004 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.770008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.770022 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:59:22.770068 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.770071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.770090 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.770325 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:22.770384 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.771568 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.772760 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.773513 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.773828 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.773925 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.773951 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.773994 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260928 21:59:22.774111 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.769619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.2.165.66:38935)
0928 21:59:22.769815 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.769820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.770077 (+ 257us) server_negotiation.cc:407] Connection header received
0928 21:59:22.770257 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.770261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.770277 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.770364 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:22.770480 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.771429 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.772894 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.773597 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.773658 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.773738 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.773985 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.774028 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260928 21:59:22.774324 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.769501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:38935 (local address 127.0.0.1:38478)
0928 21:59:22.770007 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.770018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.770106 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.770361 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.770362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.770364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:22.770425 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.770427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.771519 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.771520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.772806 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.772809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.773769 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.773770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.773836 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.773839 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.774217 (+ 378us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.774262 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":102,"threads_started":1}
I20260928 21:59:22.773782 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.769700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:40909 (local address 127.0.0.1:52804)
0928 21:59:22.770195 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.770204 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.770257 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.770488 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.770489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.770491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:22.770526 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.770528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.771712 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.771713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.772832 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.772833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.773518 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.773519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.773575 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.773580 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.773647 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.773694 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":58,"threads_started":1}
I20260928 21:59:22.774828 9973 tablet_service.cc:1467] Processing CreateTablet for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7 (DEFAULT_TABLE table=test-table [id=db070bd8fc64455089b94908a33a7b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:22.775032 9701 tablet_service.cc:1467] Processing CreateTablet for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7 (DEFAULT_TABLE table=test-table [id=db070bd8fc64455089b94908a33a7b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:22.775216 9973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:22.775328 9701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:22.779421 10067 tablet_bootstrap.cc:492] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Bootstrap starting.
I20260928 21:59:22.779767 9838 tablet_service.cc:1467] Processing CreateTablet for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7 (DEFAULT_TABLE table=test-table [id=db070bd8fc64455089b94908a33a7b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:22.780133 9838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:22.780169 10067 tablet_bootstrap.cc:654] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:22.780478 10067 log.cc:824] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:22.781352 10067 tablet_bootstrap.cc:492] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: No bootstrap required, opened a new log
I20260928 21:59:22.781410 10067 ts_tablet_manager.cc:1397] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:59:22.783190 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.779549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.2.165.124:43475)
0928 21:59:22.779661 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.779663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.779673 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:22.779716 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.779717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.779743 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.779817 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:22.780004 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.780782 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.782162 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.782769 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.782861 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.782936 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.783046 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.783096 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 21:59:22.783190 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.779441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.0.0.1:57392)
0928 21:59:22.779593 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.779604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.779646 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.779837 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.779838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.779840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:22.779883 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.779885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.780932 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.780933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.782057 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.782061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.782874 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.782875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.782952 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.782957 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.783032 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.783094 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 21:59:22.783468 10069 tablet_bootstrap.cc:492] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Bootstrap starting.
I20260928 21:59:22.783582 9598 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 21:59:22.783643 10067 raft_consensus.cc:348] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.783851 10067 raft_consensus.cc:374] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:22.783885 10067 raft_consensus.cc:729] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db60bde3e3904dddbb09dbe0f4ed8ae6, State: Initialized, Role: FOLLOWER
I20260928 21:59:22.784001 10067 consensus_queue.cc:260] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.784122 10069 tablet_bootstrap.cc:654] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:22.784291 10067 ts_tablet_manager.cc:1428] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:59:22.784440 10069 log.cc:824] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:22.784503 10066 tablet_bootstrap.cc:492] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Bootstrap starting.
I20260928 21:59:22.784534 9627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbf333743754a5f80314e9e2eae266f. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.784631 9627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:59:22.785185 10066 tablet_bootstrap.cc:654] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:22.785236 10069 tablet_bootstrap.cc:492] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: No bootstrap required, opened a new log
I20260928 21:59:22.785295 10069 ts_tablet_manager.cc:1397] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:22.785511 10066 log.cc:824] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:22.785811 10041 heartbeater.cc:503] Master 127.2.165.126:34319 was elected leader, sending a full tablet report...
I20260928 21:59:22.787464 10066 tablet_bootstrap.cc:492] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: No bootstrap required, opened a new log
I20260928 21:59:22.787618 10066 ts_tablet_manager.cc:1397] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:59:22.788071 10069 raft_consensus.cc:348] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.788201 10069 raft_consensus.cc:374] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:22.788223 10069 raft_consensus.cc:729] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf075309fb547d08fce67d52f331086, State: Initialized, Role: FOLLOWER
I20260928 21:59:22.788347 10069 consensus_queue.cc:260] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.788621 10069 ts_tablet_manager.cc:1428] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 21:59:22.788631 9905 heartbeater.cc:503] Master 127.2.165.126:34319 was elected leader, sending a full tablet report...
I20260928 21:59:22.790139 10066 raft_consensus.cc:348] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.790323 10066 raft_consensus.cc:374] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:22.790373 10066 raft_consensus.cc:729] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9c2ec29eb540fbae683aa694dd6c6b, State: Initialized, Role: FOLLOWER
I20260928 21:59:22.790500 10066 consensus_queue.cc:260] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.790791 10066 ts_tablet_manager.cc:1428] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 21:59:22.790894 9768 heartbeater.cc:503] Master 127.2.165.126:34319 was elected leader, sending a full tablet report...
W20260928 21:59:22.849499 9907 tablet.cc:2369] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:22.886729 10070 raft_consensus.cc:482] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:22.886873 10070 raft_consensus.cc:504] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.887214 10070 leader_election.cc:288] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bf075309fb547d08fce67d52f331086 (127.2.165.66:38935), fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
I20260928 21:59:22.890383 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.887386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:38935 (local address 127.2.165.67:58091)
0928 21:59:22.887449 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.887461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.887508 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.887668 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.887670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.887671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:22.887747 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.887751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.888481 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.888483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.889345 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.889348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.890089 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.890090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.890183 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.890186 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.890254 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.890315 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:59:22.890383 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.887471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:58091 (local address 127.2.165.66:38935)
0928 21:59:22.887541 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.887543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.887553 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:22.887580 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.887581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.887592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.887663 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:22.887835 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.888384 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.889452 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.889996 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.890089 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.890100 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.890260 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.890301 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:59:22.890579 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.887631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:40909 (local address 127.2.165.67:56063)
0928 21:59:22.887679 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:22.887689 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:22.887724 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:22.887925 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:22.887926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:22.887927 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:22.887997 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.887999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.888636 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.888637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.889382 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:22.889384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.890258 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.890259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:22.890377 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.890380 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:22.890453 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:59:22.890499 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 21:59:22.890638 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.887676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:56063 (local address 127.2.165.65:40909)
0928 21:59:22.887722 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.887724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.887734 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:22.887804 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.887805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.887815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.887916 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:22.888091 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.888551 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.889554 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.890152 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.890258 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.890266 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.890458 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.890560 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:59:22.890988 9858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d117b1d6d294d37a4c0fe566c3cbfb7" candidate_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf075309fb547d08fce67d52f331086" is_pre_election: true
I20260928 21:59:22.891000 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d117b1d6d294d37a4c0fe566c3cbfb7" candidate_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" is_pre_election: true
I20260928 21:59:22.891168 9721 raft_consensus.cc:2454] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db60bde3e3904dddbb09dbe0f4ed8ae6 in term 0.
I20260928 21:59:22.891180 9858 raft_consensus.cc:2454] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db60bde3e3904dddbb09dbe0f4ed8ae6 in term 0.
I20260928 21:59:22.891431 9929 leader_election.cc:302] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf075309fb547d08fce67d52f331086, db60bde3e3904dddbb09dbe0f4ed8ae6; no voters:
I20260928 21:59:22.891577 10070 raft_consensus.cc:2781] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:22.891635 10070 raft_consensus.cc:482] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:22.891671 10070 raft_consensus.cc:3037] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:22.892429 10070 raft_consensus.cc:504] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.892580 10070 leader_election.cc:288] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 0bf075309fb547d08fce67d52f331086 (127.2.165.66:38935), fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
I20260928 21:59:22.892774 9858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d117b1d6d294d37a4c0fe566c3cbfb7" candidate_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf075309fb547d08fce67d52f331086"
I20260928 21:59:22.892830 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d117b1d6d294d37a4c0fe566c3cbfb7" candidate_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b"
I20260928 21:59:22.892864 9858 raft_consensus.cc:3037] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:22.892943 9721 raft_consensus.cc:3037] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:22.893620 9858 raft_consensus.cc:2454] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db60bde3e3904dddbb09dbe0f4ed8ae6 in term 1.
I20260928 21:59:22.893725 9721 raft_consensus.cc:2454] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db60bde3e3904dddbb09dbe0f4ed8ae6 in term 1.
I20260928 21:59:22.893784 9929 leader_election.cc:302] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf075309fb547d08fce67d52f331086, db60bde3e3904dddbb09dbe0f4ed8ae6; no voters:
I20260928 21:59:22.893869 10070 raft_consensus.cc:2781] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:22.894037 10070 raft_consensus.cc:686] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 LEADER]: Becoming Leader. State: Replica: db60bde3e3904dddbb09dbe0f4ed8ae6, State: Running, Role: LEADER
I20260928 21:59:22.894163 10070 consensus_queue.cc:237] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:22.895162 8803 catalog_manager.cc:5495] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:22.895893 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: NON_VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: true } } }
I20260928 21:59:22.895996 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:22.896497 9552 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: attempting to promote NON_VOTER 98c59779fe4f47c09127ff1dad13c909 to VOTER
I20260928 21:59:22.896651 9552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:22.897130 9598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 LEARNER]: Refusing update from remote peer 4cbf333743754a5f80314e9e2eae266f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 21:59:22.897154 9411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Refusing update from remote peer 4cbf333743754a5f80314e9e2eae266f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 21:59:22.897305 10083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 21:59:22.897445 10084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 21:59:22.898202 9411 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.898259 9598 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.899201 9627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.899283 9627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:59:22.898099 10081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.899415 10085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.899510 10085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:59:22.900152 9552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.900185 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.900200 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cbf333743754a5f80314e9e2eae266f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:22.900254 9552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:22.900262 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:22.900283 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:22.919137 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.916614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.2.165.67:42575)
0928 21:59:22.916706 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.916708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.916717 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:22.916792 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.916793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.916811 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.916893 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:22.917032 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.917520 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.918294 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.918513 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.918605 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.918784 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:22.918889 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:22.918991 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.919019 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.919041 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:59:22.921198 9858 raft_consensus.cc:1261] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 1 FOLLOWER]: Refusing update from remote peer db60bde3e3904dddbb09dbe0f4ed8ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:22.921612 9721 raft_consensus.cc:1261] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Refusing update from remote peer db60bde3e3904dddbb09dbe0f4ed8ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:22.921748 10070 consensus_queue.cc:1035] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:22.921993 10079 consensus_queue.cc:1035] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:22.926517 10088 mvcc.cc:205] Tried to move back new op lower bound from 7334431796921761792 to 7334431796815196160. Current Snapshot: MvccSnapshot[applied={T|T < 7334431796921761792 or (T in {7334431796921761792})}]
I20260928 21:59:22.927618 8802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:22.927735 8802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:22.929263 8802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cbf333743754a5f80314e9e2eae266f: Sending DeleteTablet for 3 replicas of tablet 4d117b1d6d294d37a4c0fe566c3cbfb7
I20260928 21:59:22.929648 9838 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:22 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:38478
I20260928 21:59:22.929708 9973 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:22 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:43328
I20260928 21:59:22.929733 9701 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d117b1d6d294d37a4c0fe566c3cbfb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:22 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:52804
I20260928 21:59:22.930071 10096 tablet_replica.cc:330] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: stopping tablet replica
I20260928 21:59:22.930181 10096 raft_consensus.cc:2229] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:22.930294 10096 raft_consensus.cc:2258] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:22.930464 10097 tablet_replica.cc:330] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: stopping tablet replica
I20260928 21:59:22.930552 10097 raft_consensus.cc:2229] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:22.930663 10097 raft_consensus.cc:2258] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:22.930814 10098 tablet_replica.cc:330] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: stopping tablet replica
I20260928 21:59:22.930912 10098 raft_consensus.cc:2229] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:22.931025 10098 raft_consensus.cc:2258] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:22.931380 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 8771
I20260928 21:59:22.931596 10097 ts_tablet_manager.cc:1905] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:22.932037 10098 ts_tablet_manager.cc:1905] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:22.932789 10096 ts_tablet_manager.cc:1905] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:22.934803 10096 ts_tablet_manager.cc:1918] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:22.934937 10096 log.cc:1196] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/4d117b1d6d294d37a4c0fe566c3cbfb7
I20260928 21:59:22.935194 10098 ts_tablet_manager.cc:1918] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:22.935273 10098 log.cc:1196] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/4d117b1d6d294d37a4c0fe566c3cbfb7
I20260928 21:59:22.935387 10096 ts_tablet_manager.cc:1939] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P 0bf075309fb547d08fce67d52f331086: Deleting consensus metadata
I20260928 21:59:22.935570 10098 ts_tablet_manager.cc:1939] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P fc9c2ec29eb540fbae683aa694dd6c6b: Deleting consensus metadata
I20260928 21:59:22.936590 10097 ts_tablet_manager.cc:1918] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:22.936688 10097 log.cc:1196] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/4d117b1d6d294d37a4c0fe566c3cbfb7
I20260928 21:59:22.936949 10097 ts_tablet_manager.cc:1939] T 4d117b1d6d294d37a4c0fe566c3cbfb7 P db60bde3e3904dddbb09dbe0f4ed8ae6: Deleting consensus metadata
W20260928 21:59:22.937959 9655 connection.cc:535] server connection from 127.0.0.1:52804 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:59:22.937978 9793 connection.cc:535] server connection from 127.0.0.1:38478 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:59:22.938177 9929 connection.cc:535] server connection from 127.0.0.1:43328 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:59:22.948895 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.930544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.2.165.124:43475)
0928 21:59:22.930588 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.930591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.938924 (+ 8333us) server_negotiation.cc:407] Connection header received
0928 21:59:22.938990 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.938991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.939003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.939054 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.940229 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.941075 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.942164 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.942471 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.942529 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.944586 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.944587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.944588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.944595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.947380 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.947762 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.947763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.947763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.947842 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.948073 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.948074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.948075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.948229 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.948328 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.948704 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.948789 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:59:22.948987 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.940091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.2.165.125:33015)
0928 21:59:22.940134 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.940136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.940367 (+ 231us) server_negotiation.cc:407] Connection header received
0928 21:59:22.940476 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.940477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.940493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.940523 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.941854 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.942589 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.943540 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.943855 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.943895 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.945446 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.945448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.945448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.945455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.947242 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.948049 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.948050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.948050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.948106 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.948362 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.948363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.948363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.948502 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.948633 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.948810 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.948891 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:59:22.949980 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:22.939878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.2.165.124:43475)
0928 21:59:22.939930 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:22.939932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:22.940387 (+ 455us) server_negotiation.cc:407] Connection header received
0928 21:59:22.940513 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:22.940514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:22.940554 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:22.940585 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:22.941517 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.942295 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.943407 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:22.943702 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:22.943741 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:22.945357 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:22.945358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:22.945358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:22.945366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:22.947141 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.949126 (+ 1985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.949127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.949128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.949191 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:22.949388 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:22.949388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:22.949389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:22.949514 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:22.949606 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:22.949791 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 21:59:22.949877 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:59:23.927135 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:23.926809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:53565)
0928 21:59:23.926971 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:23.927069 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260928 21:59:23.927518 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:23.927293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:36835)
0928 21:59:23.927400 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:23.927473 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
W20260928 21:59:23.927578 9768 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:34319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
W20260928 21:59:23.927915 9905 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:34319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:23.928200 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:23.927973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:42641)
0928 21:59:23.928077 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:23.928154 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":65,"threads_started":1}
W20260928 21:59:23.928366 10041 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:34319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:24.429760 10110 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cbf333743754a5f80314e9e2eae266f)
I20260928 21:59:24.429929 10111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cbf333743754a5f80314e9e2eae266f)
I20260928 21:59:24.429948 10110 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:24.430087 10111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:24.430303 10110 leader_election.cc:288] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319), 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124:43475)
I20260928 21:59:24.430433 10111 leader_election.cc:288] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319), e0c923da36c948d39913ba44657480cd (127.2.165.125:33015)
I20260928 21:59:24.430685 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.430479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53642)
0928 21:59:24.430589 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.430639 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":46,"threads_started":1}
W20260928 21:59:24.430809 9388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:24.430903 10114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.430526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53650)
0928 21:59:24.430813 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.430864 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":43,"threads_started":1}
W20260928 21:59:24.431021 9575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:24.431317 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.431186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53660)
0928 21:59:24.431246 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.431273 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 21:59:24.431408 10114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.431292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53664)
0928 21:59:24.431342 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.431368 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:59:24.431483 9575 leader_election.cc:334] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319): Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
W20260928 21:59:24.431527 9388 leader_election.cc:334] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319): Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:24.434547 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.430480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:43475 (local address 127.0.0.1:57416)
0928 21:59:24.430645 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.430657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:24.430716 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:24.431496 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:24.431497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:24.431499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:24.431548 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:24.431551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.432392 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.432392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:24.433633 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:24.433636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.434259 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.434260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:24.434325 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:24.434330 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:59:24.434411 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:59:24.434457 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":50,"threads_started":1}
I20260928 21:59:24.434653 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.430714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.2.165.124:43475)
0928 21:59:24.431358 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:24.431360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:24.431367 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:59:24.431388 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:24.431389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:24.431398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:24.431614 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:24.431649 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.432332 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.433752 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.434184 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.434247 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:24.434259 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:24.434431 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:59:24.434558 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":42,"threads_started":1}
I20260928 21:59:24.434764 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.430814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.2.165.125:33015)
0928 21:59:24.430903 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:24.430905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:24.430915 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:24.431215 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:24.431216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:24.431228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:24.431309 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:24.431501 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.432221 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.433836 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.434315 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.434379 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:24.434407 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:24.434526 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:59:24.434567 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":39,"threads_started":1}
I20260928 21:59:24.434839 9598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0c923da36c948d39913ba44657480cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98c59779fe4f47c09127ff1dad13c909" is_pre_election: true
I20260928 21:59:24.434901 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.430658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:33015 (local address 127.0.0.1:42906)
0928 21:59:24.430821 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.430832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:24.430931 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:24.431325 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:24.431327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:24.431330 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:24.431402 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:24.431404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.432324 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.432324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:24.433633 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:24.433636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:24.434397 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:24.434398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:24.434455 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:24.434459 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:24.434512 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:59:24.434560 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":34,"threads_started":1}
I20260928 21:59:24.434967 9598 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0c923da36c948d39913ba44657480cd in term 1.
I20260928 21:59:24.435117 9385 leader_election.cc:302] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98c59779fe4f47c09127ff1dad13c909, e0c923da36c948d39913ba44657480cd; no voters: 4cbf333743754a5f80314e9e2eae266f
I20260928 21:59:24.435155 9411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c59779fe4f47c09127ff1dad13c909" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0c923da36c948d39913ba44657480cd" is_pre_election: true
I20260928 21:59:24.435235 10110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:59:24.435281 9411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c59779fe4f47c09127ff1dad13c909 in term 1.
I20260928 21:59:24.435316 10110 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cbf333743754a5f80314e9e2eae266f)
I20260928 21:59:24.435349 10110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:59:24.435436 9575 leader_election.cc:302] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98c59779fe4f47c09127ff1dad13c909, e0c923da36c948d39913ba44657480cd; no voters: 4cbf333743754a5f80314e9e2eae266f
I20260928 21:59:24.435571 10111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:59:24.435618 10111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cbf333743754a5f80314e9e2eae266f)
I20260928 21:59:24.435648 10111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:59:24.436192 10110 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:24.436344 10110 leader_election.cc:288] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [CANDIDATE]: Term 2 election: Requested vote from peers 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319), 98c59779fe4f47c09127ff1dad13c909 (127.2.165.124:43475)
I20260928 21:59:24.436355 10111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:24.436504 10111 leader_election.cc:288] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 2 election: Requested vote from peers 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319), e0c923da36c948d39913ba44657480cd (127.2.165.125:33015)
I20260928 21:59:24.436520 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.436424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53678)
0928 21:59:24.436465 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.436492 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:59:24.436545 9598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0c923da36c948d39913ba44657480cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98c59779fe4f47c09127ff1dad13c909"
I20260928 21:59:24.436611 9598 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e0c923da36c948d39913ba44657480cd in current term 2: Already voted for candidate 98c59779fe4f47c09127ff1dad13c909 in this term.
I20260928 21:59:24.436718 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.436581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53682)
0928 21:59:24.436614 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.436640 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 21:59:24.436718 9411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c59779fe4f47c09127ff1dad13c909" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0c923da36c948d39913ba44657480cd"
I20260928 21:59:24.436790 9411 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 98c59779fe4f47c09127ff1dad13c909 in current term 2: Already voted for candidate e0c923da36c948d39913ba44657480cd in this term.
I20260928 21:59:24.437091 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.436981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53696)
0928 21:59:24.437040 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.437066 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 21:59:24.437168 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.437058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53708)
0928 21:59:24.437109 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.437137 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 21:59:24.437170 9575 leader_election.cc:334] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319): Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:24.437206 9575 leader_election.cc:302] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98c59779fe4f47c09127ff1dad13c909; no voters: 4cbf333743754a5f80314e9e2eae266f, e0c923da36c948d39913ba44657480cd
W20260928 21:59:24.437251 9388 leader_election.cc:334] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319): Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:24.437265 10111 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:59:24.437300 9388 leader_election.cc:302] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0c923da36c948d39913ba44657480cd; no voters: 4cbf333743754a5f80314e9e2eae266f, 98c59779fe4f47c09127ff1dad13c909
I20260928 21:59:24.437358 10110 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:59:24.928155 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.927910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:57267)
0928 21:59:24.928044 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.928091 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260928 21:59:24.928656 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.928424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:57527)
0928 21:59:24.928569 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.928608 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260928 21:59:24.928848 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:24.928624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:55179)
0928 21:59:24.928767 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:24.928801 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260928 21:59:25.929001 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:25.928713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:43241)
0928 21:59:25.928843 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:25.928908 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260928 21:59:25.929348 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:25.929124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:38017)
0928 21:59:25.929266 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:25.929305 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260928 21:59:25.929376 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:25.929168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:46055)
0928 21:59:25.929293 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:25.929329 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260928 21:59:26.425048 10126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:26.425182 10126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:26.425405 10126 leader_election.cc:288] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319), e0c923da36c948d39913ba44657480cd (127.2.165.125:33015)
I20260928 21:59:26.425792 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.425587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53952)
0928 21:59:26.425724 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.425756 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":39,"threads_started":1}
I20260928 21:59:26.425876 9411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c59779fe4f47c09127ff1dad13c909" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0c923da36c948d39913ba44657480cd" is_pre_election: true
I20260928 21:59:26.426005 9411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c59779fe4f47c09127ff1dad13c909 in term 2.
I20260928 21:59:26.426263 9575 leader_election.cc:302] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98c59779fe4f47c09127ff1dad13c909, e0c923da36c948d39913ba44657480cd; no voters:
I20260928 21:59:26.426263 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.426155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53964)
0928 21:59:26.426196 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.426241 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:59:26.426376 9575 leader_election.cc:334] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319): Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:26.426421 10126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 21:59:26.426486 10126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:26.426600 10126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:59:26.427315 10126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:26.427456 10126 leader_election.cc:288] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 3 election: Requested vote from peers 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319), e0c923da36c948d39913ba44657480cd (127.2.165.125:33015)
I20260928 21:59:26.427752 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.427559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53968)
0928 21:59:26.427709 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.427732 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 21:59:26.427788 9411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98c59779fe4f47c09127ff1dad13c909" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0c923da36c948d39913ba44657480cd"
I20260928 21:59:26.427882 9411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:59:26.428161 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.428059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:53972)
0928 21:59:26.428106 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.428132 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:59:26.428233 9575 leader_election.cc:334] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319): Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
I20260928 21:59:26.428737 9411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c59779fe4f47c09127ff1dad13c909 in term 3.
I20260928 21:59:26.428913 9575 leader_election.cc:302] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98c59779fe4f47c09127ff1dad13c909, e0c923da36c948d39913ba44657480cd; no voters: 4cbf333743754a5f80314e9e2eae266f
I20260928 21:59:26.429086 10126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:59:26.429227 10126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [term 3 LEADER]: Becoming Leader. State: Replica: 98c59779fe4f47c09127ff1dad13c909, State: Running, Role: LEADER
I20260928 21:59:26.429308 10126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } }
I20260928 21:59:26.429648 10129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c59779fe4f47c09127ff1dad13c909. Latest consensus state: current_term: 3 leader_uuid: "98c59779fe4f47c09127ff1dad13c909" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:26.429745 10129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:26.429908 10131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:26.430346 10131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:26.430441 10131 catalog_manager.cc:1259] Loaded cluster ID: b3f89f07593f487d8dd67a400cac3541
I20260928 21:59:26.430475 10131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:26.430660 10131 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:26.430753 10131 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Loaded TSK: 0
I20260928 21:59:26.430892 10131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:59:26.441598 9582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:26.443224 9582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:26.444376 9411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [term 3 FOLLOWER]: Refusing update from remote peer 98c59779fe4f47c09127ff1dad13c909: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 21:59:26.444410 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.444304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:54000)
0928 21:59:26.444342 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.444372 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 21:59:26.444773 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.444608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.0.0.1:54014)
0928 21:59:26.444722 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.444750 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 21:59:26.444768 10129 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 21:59:26.444907 9575 consensus_peers.cc:487] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 -> Peer 4cbf333743754a5f80314e9e2eae266f (127.2.165.126:34319): Couldn't send request to peer 4cbf333743754a5f80314e9e2eae266f. Status: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:59:26.445429 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c59779fe4f47c09127ff1dad13c909. Latest consensus state: current_term: 3 leader_uuid: "98c59779fe4f47c09127ff1dad13c909" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:26.445552 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:59:26.446066 10132 mvcc.cc:205] Tried to move back new op lower bound from 7334431811353907200 to 7334431811294941184. Current Snapshot: MvccSnapshot[applied={T|T < 7334431811353907200}]
I20260928 21:59:26.446247 10134 mvcc.cc:205] Tried to move back new op lower bound from 7334431811353907200 to 7334431811294941184. Current Snapshot: MvccSnapshot[applied={T|T < 7334431811353907200}]
I20260928 21:59:26.446213 10129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "98c59779fe4f47c09127ff1dad13c909" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:26.446300 10129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:26.446213 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "98c59779fe4f47c09127ff1dad13c909" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4cbf333743754a5f80314e9e2eae266f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34319 } } peers { permanent_uuid: "e0c923da36c948d39913ba44657480cd" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 33015 } attrs { promote: false } } peers { permanent_uuid: "98c59779fe4f47c09127ff1dad13c909" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 43475 } attrs { promote: false } } }
I20260928 21:59:26.446532 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c923da36c948d39913ba44657480cd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:59:26.451738 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.447882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:38935 (local address 127.0.0.1:38486)
0928 21:59:26.448091 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.448104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:26.448154 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:26.448546 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:26.448547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:26.448549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:26.448626 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.448628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.449606 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.449606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.450779 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.450780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.451449 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.451449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.451531 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.451537 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:59:26.451592 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:59:26.451641 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":49}
I20260928 21:59:26.451761 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.447961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.2.165.66:38935)
0928 21:59:26.448289 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:26.448291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:26.448299 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:59:26.448330 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:26.448330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:26.448339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:26.448390 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:26.448826 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.449520 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.450875 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.451328 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.451362 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.451371 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:26.451627 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 21:59:26.451676 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":61,"threads_started":1}
I20260928 21:59:26.451738 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.447969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:40909 (local address 127.0.0.1:52806)
0928 21:59:26.448134 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.448146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:26.448198 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:26.448847 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:26.448848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:26.448851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:26.448890 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.448892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.449815 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.449816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.450707 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.450709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.451446 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.451447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.451531 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.451539 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:59:26.451592 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:59:26.451644 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260928 21:59:26.452035 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.448029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.2.165.67:42575)
0928 21:59:26.448454 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:26.448456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:26.448462 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:59:26.448482 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:26.448482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:26.448490 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:26.448561 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:26.448756 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.449618 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.451065 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.451667 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.451734 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.451746 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:26.451925 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 21:59:26.451966 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":41,"threads_started":1}
I20260928 21:59:26.452195 9838 tablet_service.cc:1467] Processing CreateTablet for tablet d295f22fb1c74f0d9faf08c696017f2e (DEFAULT_TABLE table=test-table [id=e80307f98fbe4a1594ba21d2d2cc7e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:26.452368 9838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d295f22fb1c74f0d9faf08c696017f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:26.452368 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.448126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.2.165.65:40909)
0928 21:59:26.448668 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:26.448695 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:26.448703 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:59:26.448728 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:26.448729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:26.448740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:26.448801 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:26.448965 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.449724 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.450790 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.451356 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.451437 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.451449 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:26.452197 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 21:59:26.452287 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":49,"threads_started":1}
I20260928 21:59:26.452579 9701 tablet_service.cc:1467] Processing CreateTablet for tablet d295f22fb1c74f0d9faf08c696017f2e (DEFAULT_TABLE table=test-table [id=e80307f98fbe4a1594ba21d2d2cc7e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:26.452747 9701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d295f22fb1c74f0d9faf08c696017f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:26.452869 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.447964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:42575 (local address 127.0.0.1:43350)
0928 21:59:26.448135 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.448146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:26.448198 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:26.448557 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:26.448558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:26.448559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:26.448626 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.448661 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.449699 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.449700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.450990 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.450992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.451780 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.451781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.451843 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.451847 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:26.452102 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 21:59:26.452793 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":49,"thread_start_us":74,"threads_started":1}
I20260928 21:59:26.453292 9973 tablet_service.cc:1467] Processing CreateTablet for tablet d295f22fb1c74f0d9faf08c696017f2e (DEFAULT_TABLE table=test-table [id=e80307f98fbe4a1594ba21d2d2cc7e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:26.453408 9973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d295f22fb1c74f0d9faf08c696017f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:26.453950 10144 tablet_bootstrap.cc:492] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: Bootstrap starting.
I20260928 21:59:26.454351 10144 tablet_bootstrap.cc:654] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:26.454594 10143 tablet_bootstrap.cc:492] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b: Bootstrap starting.
I20260928 21:59:26.454854 10146 tablet_bootstrap.cc:492] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Bootstrap starting.
I20260928 21:59:26.454909 10144 tablet_bootstrap.cc:492] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: No bootstrap required, opened a new log
I20260928 21:59:26.454941 10143 tablet_bootstrap.cc:654] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:26.454958 10144 ts_tablet_manager.cc:1397] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:26.455121 10144 raft_consensus.cc:348] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.455217 10146 tablet_bootstrap.cc:654] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:26.455222 10144 raft_consensus.cc:374] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:26.455251 10144 raft_consensus.cc:729] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf075309fb547d08fce67d52f331086, State: Initialized, Role: FOLLOWER
I20260928 21:59:26.455325 10144 consensus_queue.cc:260] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.455497 10144 ts_tablet_manager.cc:1428] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:26.455960 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.455757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:54205)
0928 21:59:26.455880 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.455919 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":90,"threads_started":1}
I20260928 21:59:26.455967 10146 tablet_bootstrap.cc:492] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: No bootstrap required, opened a new log
I20260928 21:59:26.456012 10146 ts_tablet_manager.cc:1397] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:26.456154 10146 raft_consensus.cc:348] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.456192 9904 heartbeater.cc:503] Master 127.2.165.124:43475 was elected leader, sending a full tablet report...
I20260928 21:59:26.456257 10146 raft_consensus.cc:374] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:26.456300 10146 raft_consensus.cc:729] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db60bde3e3904dddbb09dbe0f4ed8ae6, State: Initialized, Role: FOLLOWER
I20260928 21:59:26.456312 10143 tablet_bootstrap.cc:492] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b: No bootstrap required, opened a new log
I20260928 21:59:26.456379 10143 ts_tablet_manager.cc:1397] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:26.456378 10146 consensus_queue.cc:260] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.456558 10146 ts_tablet_manager.cc:1428] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:26.456579 10143 raft_consensus.cc:348] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.456722 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.456180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:57239)
0928 21:59:26.456635 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.456664 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":66,"threads_started":1}
I20260928 21:59:26.456825 10143 raft_consensus.cc:374] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:26.456899 10143 raft_consensus.cc:729] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9c2ec29eb540fbae683aa694dd6c6b, State: Initialized, Role: FOLLOWER
I20260928 21:59:26.457037 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.456848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:46731)
0928 21:59:26.456963 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.457002 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":73,"threads_started":1}
I20260928 21:59:26.457067 10143 consensus_queue.cc:260] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.457089 10040 heartbeater.cc:503] Master 127.2.165.124:43475 was elected leader, sending a full tablet report...
I20260928 21:59:26.457276 10143 ts_tablet_manager.cc:1428] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:59:26.457608 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.457408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:42605)
0928 21:59:26.457553 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.457580 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260928 21:59:26.457721 9767 heartbeater.cc:503] Master 127.2.165.124:43475 was elected leader, sending a full tablet report...
I20260928 21:59:26.457855 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.457443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:57911)
0928 21:59:26.457779 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.457816 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":61,"threads_started":1}
I20260928 21:59:26.458353 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.458194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:50741)
0928 21:59:26.458299 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.458322 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260928 21:59:26.496035 10043 tablet.cc:2369] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:26.635118 10149 raft_consensus.cc:482] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:26.635254 10149 raft_consensus.cc:504] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.635637 10149 leader_election.cc:288] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67:42575), fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
I20260928 21:59:26.638867 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.635946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:43483 (local address 127.2.165.67:42575)
0928 21:59:26.636138 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:26.636140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:26.636148 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:59:26.636178 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:26.636178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:26.636207 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:26.636262 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:26.636441 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.637055 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.638132 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.638512 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.638604 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.638613 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:26.638759 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 21:59:26.638798 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 21:59:26.638861 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.635947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:51199 (local address 127.2.165.65:40909)
0928 21:59:26.635994 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:26.635996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:26.636006 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:26.636133 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:26.636134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:26.636142 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:26.636205 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:26.636378 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.636997 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.638100 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.638484 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.638568 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.638578 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:26.638754 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 21:59:26.638789 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:59:26.638875 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.635849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:40909 (local address 127.2.165.66:51199)
0928 21:59:26.635918 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.635931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:26.636027 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:26.636229 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:26.636230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:26.636232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:26.636297 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.636299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.637071 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.637072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.637995 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.637997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.638568 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.638569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.638664 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.638667 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:26.638744 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:59:26.638787 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":31}
I20260928 21:59:26.638878 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.635849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:42575 (local address 127.2.165.66:43483)
0928 21:59:26.636023 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.636032 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:26.636072 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:26.636292 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:26.636293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:26.636295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:26.636359 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.636361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.637132 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.637133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.637996 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:26.637998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.638602 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.638603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:26.638672 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.638675 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:26.638751 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:59:26.638791 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":22}
I20260928 21:59:26.639226 9994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d295f22fb1c74f0d9faf08c696017f2e" candidate_uuid: "0bf075309fb547d08fce67d52f331086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" is_pre_election: true
I20260928 21:59:26.639349 9994 raft_consensus.cc:2454] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf075309fb547d08fce67d52f331086 in term 0.
I20260928 21:59:26.639371 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d295f22fb1c74f0d9faf08c696017f2e" candidate_uuid: "0bf075309fb547d08fce67d52f331086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" is_pre_election: true
I20260928 21:59:26.639487 9721 raft_consensus.cc:2454] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf075309fb547d08fce67d52f331086 in term 0.
I20260928 21:59:26.639597 9794 leader_election.cc:302] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf075309fb547d08fce67d52f331086, db60bde3e3904dddbb09dbe0f4ed8ae6; no voters:
I20260928 21:59:26.639762 10149 raft_consensus.cc:2781] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:26.639798 10149 raft_consensus.cc:482] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:26.639813 10149 raft_consensus.cc:3037] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:26.640482 10149 raft_consensus.cc:504] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.640630 10149 leader_election.cc:288] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [CANDIDATE]: Term 1 election: Requested vote from peers db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67:42575), fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909)
I20260928 21:59:26.640787 9721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d295f22fb1c74f0d9faf08c696017f2e" candidate_uuid: "0bf075309fb547d08fce67d52f331086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b"
I20260928 21:59:26.640868 9721 raft_consensus.cc:3037] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:26.640827 9994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d295f22fb1c74f0d9faf08c696017f2e" candidate_uuid: "0bf075309fb547d08fce67d52f331086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6"
I20260928 21:59:26.640944 9994 raft_consensus.cc:3037] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:26.641471 9721 raft_consensus.cc:2454] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf075309fb547d08fce67d52f331086 in term 1.
I20260928 21:59:26.641628 9793 leader_election.cc:302] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf075309fb547d08fce67d52f331086, fc9c2ec29eb540fbae683aa694dd6c6b; no voters:
I20260928 21:59:26.641726 10149 raft_consensus.cc:2781] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:26.641827 9994 raft_consensus.cc:2454] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf075309fb547d08fce67d52f331086 in term 1.
I20260928 21:59:26.641902 10149 raft_consensus.cc:686] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf075309fb547d08fce67d52f331086, State: Running, Role: LEADER
I20260928 21:59:26.641983 10149 consensus_queue.cc:237] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } }
I20260928 21:59:26.642338 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.642140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:33241)
0928 21:59:26.642227 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.642286 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 21:59:26.642793 9584 catalog_manager.cc:5495] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bf075309fb547d08fce67d52f331086 (127.2.165.66). New cstate: current_term: 1 leader_uuid: "0bf075309fb547d08fce67d52f331086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf075309fb547d08fce67d52f331086" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:26.682113 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.679338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.2.165.66:38935)
0928 21:59:26.679441 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:26.679443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:26.679453 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:26.679494 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:26.679495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:26.679514 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:26.679599 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:26.679917 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.680450 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.681286 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:26.681494 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:26.681554 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:26.681739 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:26.681852 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:26.681913 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:26.681951 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:59:26.681970 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 21:59:26.683357 9721 raft_consensus.cc:1261] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Refusing update from remote peer 0bf075309fb547d08fce67d52f331086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:26.683494 9994 raft_consensus.cc:1261] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 FOLLOWER]: Refusing update from remote peer 0bf075309fb547d08fce67d52f331086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:26.683748 10149 consensus_queue.cc:1035] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9c2ec29eb540fbae683aa694dd6c6b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:26.683919 10157 consensus_queue.cc:1035] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db60bde3e3904dddbb09dbe0f4ed8ae6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 42575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:26.684995 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.684823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:33883)
0928 21:59:26.684912 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.684951 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 21:59:26.686092 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.685906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.65:43799)
0928 21:59:26.686028 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.686060 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 21:59:26.686693 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.685804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:47629)
0928 21:59:26.686627 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.686664 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762}
I20260928 21:59:26.687024 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.686905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.66:37253)
0928 21:59:26.686974 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.687002 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 21:59:26.687623 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.687446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:59129)
0928 21:59:26.687540 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.687587 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 21:59:26.687661 10162 mvcc.cc:205] Tried to move back new op lower bound from 7334431812332867584 to 7334431812166144000. Current Snapshot: MvccSnapshot[applied={T|T < 7334431812332867584}]
I20260928 21:59:26.688017 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:26.687873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34319 (local address 127.2.165.67:42593)
0928 21:59:26.687960 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.687987 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 21:59:26.688329 9582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:26.688431 9582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:26.689870 9582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98c59779fe4f47c09127ff1dad13c909: Sending DeleteTablet for 3 replicas of tablet d295f22fb1c74f0d9faf08c696017f2e
I20260928 21:59:26.690240 9838 tablet_service.cc:1514] Processing DeleteTablet for tablet d295f22fb1c74f0d9faf08c696017f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:26 UTC) from {username='oryx', principal='oryx/127.2.165.124@KRBTEST.COM'} at 127.0.0.1:38486
I20260928 21:59:26.690269 9701 tablet_service.cc:1514] Processing DeleteTablet for tablet d295f22fb1c74f0d9faf08c696017f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:26 UTC) from {username='oryx', principal='oryx/127.2.165.124@KRBTEST.COM'} at 127.0.0.1:52806
I20260928 21:59:26.690518 10172 tablet_replica.cc:330] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b: stopping tablet replica
I20260928 21:59:26.690587 10172 raft_consensus.cc:2229] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:26.690689 10172 raft_consensus.cc:2258] T d295f22fb1c74f0d9faf08c696017f2e P fc9c2ec29eb540fbae683aa694dd6c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:26.690734 9973 tablet_service.cc:1514] Processing DeleteTablet for tablet d295f22fb1c74f0d9faf08c696017f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:26 UTC) from {username='oryx', principal='oryx/127.2.165.124@KRBTEST.COM'} at 127.0.0.1:43350
I20260928 21:59:26.690963 10173 tablet_replica.cc:330] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: stopping tablet replica
I20260928 21:59:26.691061 10173 raft_consensus.cc:2229] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:26.691130 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 9548
I20260928 21:59:26.691154 10173 raft_consensus.cc:2258] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:26.691507 10173 ts_tablet_manager.cc:1905] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:26.693547 10173 ts_tablet_manager.cc:1918] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:26.693673 10173 log.cc:1196] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/d295f22fb1c74f0d9faf08c696017f2e
I20260928 21:59:26.693756 10171 tablet_replica.cc:330] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: stopping tablet replica
I20260928 21:59:26.693837 10171 raft_consensus.cc:2229] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:26.694015 10171 raft_consensus.cc:2258] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:26.694206 10173 ts_tablet_manager.cc:1939] T d295f22fb1c74f0d9faf08c696017f2e P db60bde3e3904dddbb09dbe0f4ed8ae6: Deleting consensus metadata
I20260928 21:59:26.694366 10171 ts_tablet_manager.cc:1905] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:26.694679 9575 catalog_manager.cc:4841] TS db60bde3e3904dddbb09dbe0f4ed8ae6 (127.2.165.67:42575): tablet d295f22fb1c74f0d9faf08c696017f2e (table test-table [id=e80307f98fbe4a1594ba21d2d2cc7e28]) successfully deleted
I20260928 21:59:26.696462 10171 ts_tablet_manager.cc:1918] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:26.696535 10171 log.cc:1196] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/d295f22fb1c74f0d9faf08c696017f2e
I20260928 21:59:26.696839 10171 ts_tablet_manager.cc:1939] T d295f22fb1c74f0d9faf08c696017f2e P 0bf075309fb547d08fce67d52f331086: Deleting consensus metadata
I20260928 21:59:26.697216 9574 catalog_manager.cc:4841] TS 0bf075309fb547d08fce67d52f331086 (127.2.165.66:38935): tablet d295f22fb1c74f0d9faf08c696017f2e (table test-table [id=e80307f98fbe4a1594ba21d2d2cc7e28]) successfully deleted
W20260928 21:59:26.699235 10140 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:26.699027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:40909 (local address 127.0.0.1:52808)
0928 21:59:26.699071 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:26.699084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:26.699152 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:26.699192 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:40909: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 21:59:26.699421 9574 catalog_manager.cc:4584] TS fc9c2ec29eb540fbae683aa694dd6c6b (127.2.165.65:40909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d295f22fb1c74f0d9faf08c696017f2e: Network error: Client connection negotiation failed: client connection to 127.2.165.65:40909: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 21:59:26.699656 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 9775
I20260928 21:59:26.705376 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 9911
I20260928 21:59:26.711344 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 9267
I20260928 21:59:26.716434 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 9455
2026-09-28T21:59:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6853 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:26 dist-test-slave-3t36 krb5kdc[10176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:26 dist-test-slave-3t36 krb5kdc[10176](info): set up 2 sockets
Sep 28 21:59:26 dist-test-slave-3t36 krb5kdc[10176](info): commencing operation
krb5kdc: starting...
W20260928 21:59:27.769331 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:27 dist-test-slave-3t36 krb5kdc[10176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab.
Entry for principal oryx/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab.
I20260928 21:59:27.896307 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42903
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab
--principal=oryx/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:42903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:27.979375 10189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:27.979611 10189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:27.979637 10189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:27.980921 10189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:27.980967 10189 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:59:27.980980 10189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:27.980993 10189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:27.981006 10189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:27.981019 10189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:27.982574 10189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:42903
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.126.keytab
--principal=oryx/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:27.982793 10189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:27.982976 10189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:27.988065 10194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:27.988029 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:27.988035 10195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:27.988368 10189 server_base.cc:1034] running on GCE node
I20260928 21:59:27.988605 10189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:27.988933 10189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:27.990092 10189 hybrid_clock.cc:648] HybridClock initialized: now 1790632767990080 us; error 34 us; skew 500 ppm
Sep 28 21:59:27 dist-test-slave-3t36 krb5kdc[10176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632767, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:28.001847 10189 init.cc:377] Logged in from keytab as oryx/127.2.165.126@KRBTEST.COM (short username oryx)
I20260928 21:59:28.002405 10189 webserver.cc:460] Webserver started at http://127.2.165.126:46831/ using document root <none> and password file <none>
I20260928 21:59:28.002660 10189 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:28.002714 10189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:28.002868 10189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:28.004029 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "e1cdb59c72c4493d84e1f10b095a8077"
format_stamp: "Formatted at 2026-09-28 21:59:27 on dist-test-slave-3t36"
server_key: "1c2b8f3da73433305c30b0bd323c2e4a"
server_key_iv: "d2708c616f4c147803a34d34da24aa38"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.004428 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "e1cdb59c72c4493d84e1f10b095a8077"
format_stamp: "Formatted at 2026-09-28 21:59:27 on dist-test-slave-3t36"
server_key: "1c2b8f3da73433305c30b0bd323c2e4a"
server_key_iv: "d2708c616f4c147803a34d34da24aa38"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.006036 10189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 21:59:28.006978 10204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.007236 10189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:28.007361 10189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "e1cdb59c72c4493d84e1f10b095a8077"
format_stamp: "Formatted at 2026-09-28 21:59:27 on dist-test-slave-3t36"
server_key: "1c2b8f3da73433305c30b0bd323c2e4a"
server_key_iv: "d2708c616f4c147803a34d34da24aa38"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.007462 10189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:28.020730 10189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:28.022853 10189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:28.023036 10189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:28.027928 10189 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:42903
I20260928 21:59:28.027963 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:42903 every 8 connection(s)
I20260928 21:59:28.028378 10189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:28.029202 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:28.031569 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Bootstrap starting.
I20260928 21:59:28.031785 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 10189
I20260928 21:59:28.031900 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:28.032076 2709 external_mini_cluster.cc:1428] Setting key 3601a5178d1e191a761a9a9718160460
I20260928 21:59:28.032393 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:28.032819 10267 log.cc:824] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:28.033672 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: No bootstrap required, opened a new log
I20260928 21:59:28.035490 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdb59c72c4493d84e1f10b095a8077" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42903 } }
I20260928 21:59:28.035678 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:28.035714 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cdb59c72c4493d84e1f10b095a8077, State: Initialized, Role: FOLLOWER
I20260928 21:59:28.035837 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdb59c72c4493d84e1f10b095a8077" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42903 } }
I20260928 21:59:28.035904 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:28.035931 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:28.035992 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:28.036660 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdb59c72c4493d84e1f10b095a8077" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42903 } }
I20260928 21:59:28.036836 10267 leader_election.cc:302] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cdb59c72c4493d84e1f10b095a8077; no voters:
I20260928 21:59:28.037024 10267 leader_election.cc:288] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:28.037065 10272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:28.037238 10272 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [term 1 LEADER]: Becoming Leader. State: Replica: e1cdb59c72c4493d84e1f10b095a8077, State: Running, Role: LEADER
I20260928 21:59:28.037302 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:28.037411 10272 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdb59c72c4493d84e1f10b095a8077" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42903 } }
I20260928 21:59:28.037716 10272 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cdb59c72c4493d84e1f10b095a8077. Latest consensus state: current_term: 1 leader_uuid: "e1cdb59c72c4493d84e1f10b095a8077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdb59c72c4493d84e1f10b095a8077" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42903 } } }
I20260928 21:59:28.037729 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1cdb59c72c4493d84e1f10b095a8077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdb59c72c4493d84e1f10b095a8077" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42903 } } }
I20260928 21:59:28.037881 10272 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:28.038151 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077 [sys.catalog]: This master's current role is: LEADER
W20260928 21:59:28.039402 10288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:59:28.040077 10288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:59:28.040148 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:28.040328 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:28.043078 10279 catalog_manager.cc:1347] Generated new cluster ID: 45837831c41e4d84b5cd3e6e7966cf66
I20260928 21:59:28.043141 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:28.044752 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.032633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.2.165.126:42903)
0928 21:59:28.032849 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.032859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.032894 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:59:28.033472 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.033482 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.033554 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.033715 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.034436 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.036210 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.037213 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.037974 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.038091 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.041202 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.041206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.041207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.041219 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.042966 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.043407 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.043410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.043411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.043510 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.043737 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.043738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.043738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.044023 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.044175 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.044344 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.044482 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260928 21:59:28.050248 10279 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:28.050948 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:28.059854 10279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Generated new TSK 0
I20260928 21:59:28.060135 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab.
Entry for principal oryx/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab.
I20260928 21:59:28.088130 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42903
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab
--principal=oryx/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:28.173580 10293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:28.173811 10293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:28.173838 10293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:28.175017 10293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:28.175058 10293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:28.175078 10293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:59:28.175096 10293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:28.176640 10293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab
--principal=oryx/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:28.176913 10293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:28.177106 10293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:28.182482 10301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:28.182523 10298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:28.182521 10299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:28.182897 10293 server_base.cc:1034] running on GCE node
I20260928 21:59:28.183140 10293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:28.183463 10293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:28.184648 10293 hybrid_clock.cc:648] HybridClock initialized: now 1790632768184614 us; error 52 us; skew 500 ppm
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:28.187435 10293 init.cc:377] Logged in from keytab as oryx/127.2.165.65@KRBTEST.COM (short username oryx)
I20260928 21:59:28.187925 10293 webserver.cc:460] Webserver started at http://127.2.165.65:39763/ using document root <none> and password file <none>
I20260928 21:59:28.188169 10293 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:28.188247 10293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:28.188400 10293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:28.189608 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "65add314088c466ea3aaf1634e73c149"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "ee659d6ae00c2a8d585b8d8b1433d7a5"
server_key_iv: "28ad130913f156d4ca5feb5784745332"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.189988 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "65add314088c466ea3aaf1634e73c149"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "ee659d6ae00c2a8d585b8d8b1433d7a5"
server_key_iv: "28ad130913f156d4ca5feb5784745332"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.191425 10293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:59:28.192332 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.192548 10293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:28.192713 10293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "65add314088c466ea3aaf1634e73c149"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "ee659d6ae00c2a8d585b8d8b1433d7a5"
server_key_iv: "28ad130913f156d4ca5feb5784745332"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.192809 10293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:28.220350 10293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:28.222946 10293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:28.223165 10293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:28.223948 10293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:28.224026 10293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.224066 10293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:28.224094 10293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:28.231372 10293 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:37757
I20260928 21:59:28.231451 10423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:37757 every 8 connection(s)
I20260928 21:59:28.231765 10293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:28.233834 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 10293
I20260928 21:59:28.233953 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:28.234170 2709 external_mini_cluster.cc:1428] Setting key c44fb740ca2600a77271a7a13e19fd8f
I20260928 21:59:28.234504 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.224407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:60027 (local address 127.2.165.126:42903)
0928 21:59:28.224484 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.224487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.224788 (+ 301us) server_negotiation.cc:407] Connection header received
0928 21:59:28.225286 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.225288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.225301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.225396 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.226026 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.226910 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.228639 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.228922 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.229019 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.231342 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.231343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.231344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.231353 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.232972 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.233397 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.233398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.233398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.233449 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.233738 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.233739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.233740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.233900 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.233982 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.234319 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.234413 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:59:28.234696 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.224399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42903 (local address 127.2.165.65:60027)
0928 21:59:28.224576 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.224619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.225185 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.225454 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.225458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.225709 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:28.225914 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.225918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.227032 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.227033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.228536 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.228540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.229047 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.229049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.229189 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.229770 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:28.229779 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:28.231220 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:28.233059 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.233065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.233067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.233326 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.233507 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.233507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.233508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.233606 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.233981 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:28.233987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:28.234163 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.234293 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.234463 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260928 21:59:28.235343 10424 heartbeater.cc:347] Connected to a master server at 127.2.165.126:42903
I20260928 21:59:28.235430 10424 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:28.235618 10424 heartbeater.cc:511] Master 127.2.165.126:42903 requested a full tablet report, sending...
I20260928 21:59:28.236214 10220 ts_manager.cc:194] Registered new tserver with Master: 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757)
I20260928 21:59:28.237234 10220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:60027
WARNING: no policy specified for oryx/127.2.165.66@KRBTEST.COM; defaulting to no policy
I20260928 21:59:28.243331 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.236317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42903 (local address 127.2.165.65:53993)
0928 21:59:28.236928 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.236940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.237000 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.237161 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.237162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.237362 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:28.237431 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.237433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.238248 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.238249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.238926 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.238928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.239367 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.239368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.239440 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.239986 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:28.239991 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:28.240306 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:28.242122 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.242122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.242123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.242314 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.242629 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.242630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.242630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.242706 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.242979 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:28.242980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:28.243040 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.243130 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.243247 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":75,"threads_started":1}
I20260928 21:59:28.243396 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.236763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:53993 (local address 127.2.165.126:42903)
0928 21:59:28.236828 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.236830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.237004 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:59:28.237051 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.237052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.237065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.237160 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.237503 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.238155 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.239024 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.239280 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.239366 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.240440 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.240441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.240442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.240449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.241989 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.242393 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.242395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.242395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.242462 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.242775 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.242776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.242776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.242907 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.242989 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.243199 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.243299 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab.
Entry for principal oryx/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab.
I20260928 21:59:28.264572 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42903
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab
--principal=oryx/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:28.349051 10433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:28.349324 10433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:28.349367 10433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:28.350636 10433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:28.350725 10433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:28.350760 10433 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:59:28.350800 10433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:28.352490 10433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab
--principal=oryx/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:28.352826 10433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:28.353082 10433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:59:28.358666 10433 server_base.cc:1034] running on GCE node
W20260928 21:59:28.358440 10439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:28.358435 10441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:28.358457 10438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:28.359058 10433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:28.359359 10433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:28.360538 10433 hybrid_clock.cc:648] HybridClock initialized: now 1790632768360534 us; error 102 us; skew 500 ppm
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:28.363215 10433 init.cc:377] Logged in from keytab as oryx/127.2.165.66@KRBTEST.COM (short username oryx)
I20260928 21:59:28.363735 10433 webserver.cc:460] Webserver started at http://127.2.165.66:33349/ using document root <none> and password file <none>
I20260928 21:59:28.363984 10433 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:28.364056 10433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:28.364212 10433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:28.365381 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "9df9466c3691431ab1979d9f7d0f55ff"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "69c3681d59c36cbd5dce25d30e4e57b3"
server_key_iv: "bbd2e16b9667750187bc7f8a40d43c34"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.365747 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "9df9466c3691431ab1979d9f7d0f55ff"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "69c3681d59c36cbd5dce25d30e4e57b3"
server_key_iv: "bbd2e16b9667750187bc7f8a40d43c34"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.367101 10433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:28.368054 10448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.368312 10433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:28.368417 10433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "9df9466c3691431ab1979d9f7d0f55ff"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "69c3681d59c36cbd5dce25d30e4e57b3"
server_key_iv: "bbd2e16b9667750187bc7f8a40d43c34"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.368515 10433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:28.387156 10433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:28.389667 10433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:28.389833 10433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:28.390429 10433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:28.390471 10433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.390496 10433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:28.390554 10433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:28.397943 10433 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:45485
I20260928 21:59:28.398031 10563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:45485 every 8 connection(s)
I20260928 21:59:28.398381 10433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:28.399910 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 10433
I20260928 21:59:28.400038 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:28.400235 2709 external_mini_cluster.cc:1428] Setting key 43e9423773e9469777e40ff924647d99
I20260928 21:59:28.401309 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.390927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:46813 (local address 127.2.165.126:42903)
0928 21:59:28.391011 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.391013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.391184 (+ 171us) server_negotiation.cc:407] Connection header received
0928 21:59:28.391677 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.391678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.391694 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.391786 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.392760 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.393608 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.394768 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.395060 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.395156 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.397776 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.397777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.397778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.397788 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.399412 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.399890 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.399892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.399892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.399959 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.400302 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.400303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.400303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.400465 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.400555 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.401088 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.401198 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:59:28.401799 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.390892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42903 (local address 127.2.165.66:46813)
0928 21:59:28.391033 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.391076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.391574 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.391880 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.391884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.392321 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:28.392635 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.392644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.393742 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.393743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.394664 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.394667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.395173 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.395174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.395315 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.395914 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:28.395923 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:28.397619 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:28.399514 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.399521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.399523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.399791 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.400049 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.400050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.400051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.400222 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.400648 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:28.400656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:28.400867 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.401020 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.401285 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 21:59:28.402452 10564 heartbeater.cc:347] Connected to a master server at 127.2.165.126:42903
I20260928 21:59:28.402653 10564 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:28.402856 10564 heartbeater.cc:511] Master 127.2.165.126:42903 requested a full tablet report, sending...
I20260928 21:59:28.403307 10220 ts_manager.cc:194] Registered new tserver with Master: 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66:45485)
I20260928 21:59:28.403810 10220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:46813
WARNING: no policy specified for oryx/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 21:59:28.410199 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.403585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42903 (local address 127.2.165.66:56011)
0928 21:59:28.403793 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.403804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.403867 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.404058 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.404059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.404300 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:28.404387 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.404390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.405213 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.405214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.405842 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.405845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.406232 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.406233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.406299 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.406876 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:28.406880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:28.407199 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:28.409014 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.409016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.409017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.409218 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.409432 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.409434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.409434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.409502 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.409812 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:28.409813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:28.409867 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.409939 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.410052 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260928 21:59:28.410284 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.403647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:56011 (local address 127.2.165.126:42903)
0928 21:59:28.403697 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.403699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.403875 (+ 176us) server_negotiation.cc:407] Connection header received
0928 21:59:28.403981 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.403982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.403993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.404052 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.404489 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.405088 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.405949 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.406139 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.406230 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.407318 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.407320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.407321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.407328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.408892 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.409306 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.409307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.409308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.409368 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.409576 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.409577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.409577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.409740 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.409822 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.410065 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.410161 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab.
Entry for principal oryx/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab.
I20260928 21:59:28.433207 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:42903
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab
--principal=oryx/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:28.518047 10573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:28.518277 10573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:28.518306 10573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:28.519517 10573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:28.519565 10573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:28.519587 10573 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:59:28.519606 10573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:28.521229 10573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab
--principal=oryx/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.165.126:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:28.521449 10573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:28.521641 10573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:28.527160 10578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:28.527218 10573 server_base.cc:1034] running on GCE node
W20260928 21:59:28.527223 10581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:28.527163 10579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:28.527565 10573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:28.527848 10573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:28.529042 10573 hybrid_clock.cc:648] HybridClock initialized: now 1790632768529037 us; error 39 us; skew 500 ppm
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:28.531834 10573 init.cc:377] Logged in from keytab as oryx/127.2.165.67@KRBTEST.COM (short username oryx)
I20260928 21:59:28.532321 10573 webserver.cc:460] Webserver started at http://127.2.165.67:34241/ using document root <none> and password file <none>
I20260928 21:59:28.532558 10573 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:28.532629 10573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:28.532816 10573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:28.533927 10573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "370b233c35a84f86a4551b3306cd0c8f"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "6684d7bbd30ff1482ad3592487ce4f94"
server_key_iv: "792ee59e2f7dfc6f4dfa1b6cb4215d96"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.534307 10573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "370b233c35a84f86a4551b3306cd0c8f"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "6684d7bbd30ff1482ad3592487ce4f94"
server_key_iv: "792ee59e2f7dfc6f4dfa1b6cb4215d96"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.535871 10573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:28.536904 10588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.537177 10573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:28.537328 10573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "370b233c35a84f86a4551b3306cd0c8f"
format_stamp: "Formatted at 2026-09-28 21:59:28 on dist-test-slave-3t36"
server_key: "6684d7bbd30ff1482ad3592487ce4f94"
server_key_iv: "792ee59e2f7dfc6f4dfa1b6cb4215d96"
server_key_version: "encryptionkey@0"
I20260928 21:59:28.537432 10573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:28.551795 10573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:28.554505 10573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:28.554690 10573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:28.555253 10573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:28.555325 10573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.555377 10573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:28.555418 10573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:28.563180 10573 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:32953
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:28.563268 10703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:32953 every 8 connection(s)
I20260928 21:59:28.563577 10573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:28.566812 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.556034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:38205 (local address 127.2.165.126:42903)
0928 21:59:28.556108 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.556111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.556351 (+ 240us) server_negotiation.cc:407] Connection header received
0928 21:59:28.557003 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.557005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.557019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.557112 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.558094 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.559098 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.560951 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.561182 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.561281 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.563788 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.563790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.563790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.563799 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.565320 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.565774 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.565775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.565776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.565841 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.566131 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.566132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.566133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.566258 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.566339 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.566630 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.566717 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:59:28.567019 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.556001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42903 (local address 127.2.165.67:38205)
0928 21:59:28.556207 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.556251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.556868 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.557202 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.557207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.557610 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:28.557940 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.557947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.559238 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.559239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.560843 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.560847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.561290 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.561292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.561435 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.562058 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:28.562068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:28.563675 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:28.565427 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.565435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.565437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.565688 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.565924 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.565925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.565925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.566044 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.566328 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:28.566335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:28.566490 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.566604 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.566775 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":133,"threads_started":1}
I20260928 21:59:28.567524 10704 heartbeater.cc:347] Connected to a master server at 127.2.165.126:42903
I20260928 21:59:28.567646 10704 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:28.567831 10704 heartbeater.cc:511] Master 127.2.165.126:42903 requested a full tablet report, sending...
I20260928 21:59:28.568261 10221 ts_manager.cc:194] Registered new tserver with Master: 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:28.568820 10221 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:38205
I20260928 21:59:28.568826 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 10573
I20260928 21:59:28.568915 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:28.569087 2709 external_mini_cluster.cc:1428] Setting key 4caefd91f925db6200f9730eade465be
I20260928 21:59:28.570026 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:59:28.575501 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.568656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:60149 (local address 127.2.165.126:42903)
0928 21:59:28.568723 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.568725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.568826 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:59:28.568887 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.568888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.568900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.568933 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.569441 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.570153 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.570841 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.571053 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.571149 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.572185 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.572186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.572187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.572194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.573974 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.574492 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.574493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.574493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.574553 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.574861 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.574861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.574862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.574982 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.575110 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.575285 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.575372 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:59:28.575498 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.568566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42903 (local address 127.2.165.67:60149)
0928 21:59:28.568755 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.568765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.568827 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.569033 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.569034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.569269 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:28.569350 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.569352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.570231 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.570232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.570753 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.570755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.571152 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.571153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.571226 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.571787 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:28.571792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:28.572082 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:28.574103 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.574104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.574105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.574389 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.574711 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:28.574712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:28.574712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:28.574788 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:28.575100 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:28.575101 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:28.575155 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.575259 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.575374 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:59:28.594051 10270 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:28.588248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.2.165.126:42903)
0928 21:59:28.588318 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.588320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.588328 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:59:28.588353 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.588354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.588363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.588430 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.589219 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.589734 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.590490 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.590717 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.590778 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.593940 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42540: BlockingRecv error: recv got EOF from 127.0.0.1:42540 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 21:59:28 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:28.602229 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.594935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.2.165.126:42903)
0928 21:59:28.595042 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.595044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.595053 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:28.595099 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.595099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.595108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.595162 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:28.595708 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.596277 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.597061 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.597261 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.597318 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.599635 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:28.599637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:28.599637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:28.599644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:28.601139 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.601458 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.601459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.601459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.601516 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:28.601727 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:28.601727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:28.601728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:28.601839 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:28.601914 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.602020 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.602096 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:59:28.604137 10221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:28.605872 10221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:28.613724 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.608696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:37757 (local address 127.0.0.1:35790)
0928 21:59:28.608889 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.608934 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.609000 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.610016 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.610017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.610020 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:28.610070 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.610072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.611347 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.611348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.612521 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.612523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.613343 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.613344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.613437 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.613452 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.613526 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.613586 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":56,"thread_start_us":71,"threads_started":1}
I20260928 21:59:28.613914 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.609197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.2.165.67:32953)
0928 21:59:28.609394 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.609399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.609410 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:28.609469 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.609471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.609484 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.609582 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:28.609771 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.610986 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.612505 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.613278 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.613350 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.613426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.613764 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.613803 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260928 21:59:28.613981 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.608698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:45485 (local address 127.0.0.1:47246)
0928 21:59:28.609937 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.609948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.610043 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.610304 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.610305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.610307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:28.610383 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.610385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.611818 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.611819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.612782 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.612784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.613756 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.613757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.613812 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.613815 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.613864 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.613903 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260928 21:59:28.613705 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.609233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.2.165.65:37757)
0928 21:59:28.609808 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.609812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.609822 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:28.609860 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.609862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.609878 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.610147 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:28.610208 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.611250 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.612616 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.613243 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.613334 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.613418 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.613552 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.613589 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":80,"threads_started":1}
I20260928 21:59:28.614287 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.609117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.2.165.66:45485)
0928 21:59:28.609719 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.609725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.610011 (+ 286us) server_negotiation.cc:407] Connection header received
0928 21:59:28.610139 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.610143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.610159 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.610212 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:28.610457 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.611723 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.612850 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.613595 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.613655 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.613722 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.614145 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.614189 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":91,"threads_started":1}
I20260928 21:59:28.614415 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.608909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:32953 (local address 127.0.0.1:35902)
0928 21:59:28.609295 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.609306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.609366 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.609597 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.609601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.609603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:28.609681 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.609683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.611080 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.611081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.612412 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.612415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.613461 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.613462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.613537 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.613542 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.613594 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.613647 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260928 21:59:28.614723 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 529f76082bce41ba8020cc0cb2ad4d73 (DEFAULT_TABLE table=test-table [id=33f57b32c6994cd8a3a4690e17afd8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:28.615092 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f76082bce41ba8020cc0cb2ad4d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:28.615039 10638 tablet_service.cc:1467] Processing CreateTablet for tablet 529f76082bce41ba8020cc0cb2ad4d73 (DEFAULT_TABLE table=test-table [id=33f57b32c6994cd8a3a4690e17afd8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:28.615371 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f76082bce41ba8020cc0cb2ad4d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:28.618330 10732 tablet_bootstrap.cc:492] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Bootstrap starting.
I20260928 21:59:28.618568 10733 tablet_bootstrap.cc:492] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Bootstrap starting.
I20260928 21:59:28.618861 10497 tablet_service.cc:1467] Processing CreateTablet for tablet 529f76082bce41ba8020cc0cb2ad4d73 (DEFAULT_TABLE table=test-table [id=33f57b32c6994cd8a3a4690e17afd8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:28.619035 10732 tablet_bootstrap.cc:654] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:28.619158 10733 tablet_bootstrap.cc:654] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:28.619236 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f76082bce41ba8020cc0cb2ad4d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:28.619364 10732 log.cc:824] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:28.619503 10733 log.cc:824] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:28.620280 10732 tablet_bootstrap.cc:492] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: No bootstrap required, opened a new log
I20260928 21:59:28.620360 10732 ts_tablet_manager.cc:1397] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:28.620313 10733 tablet_bootstrap.cc:492] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: No bootstrap required, opened a new log
I20260928 21:59:28.620522 10733 ts_tablet_manager.cc:1397] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:28.622545 10736 tablet_bootstrap.cc:492] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Bootstrap starting.
I20260928 21:59:28.622455 10732 raft_consensus.cc:348] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.622625 10732 raft_consensus.cc:374] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:28.622680 10732 raft_consensus.cc:729] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370b233c35a84f86a4551b3306cd0c8f, State: Initialized, Role: FOLLOWER
I20260928 21:59:28.622561 10733 raft_consensus.cc:348] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.622802 10732 consensus_queue.cc:260] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.622876 10733 raft_consensus.cc:374] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:28.622918 10733 raft_consensus.cc:729] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65add314088c466ea3aaf1634e73c149, State: Initialized, Role: FOLLOWER
I20260928 21:59:28.623059 10732 ts_tablet_manager.cc:1428] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:59:28.623052 10733 consensus_queue.cc:260] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.623183 10704 heartbeater.cc:503] Master 127.2.165.126:42903 was elected leader, sending a full tablet report...
I20260928 21:59:28.623211 10736 tablet_bootstrap.cc:654] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:28.623329 10733 ts_tablet_manager.cc:1428] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 21:59:28.623535 10424 heartbeater.cc:503] Master 127.2.165.126:42903 was elected leader, sending a full tablet report...
I20260928 21:59:28.623561 10736 log.cc:824] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:28.624496 10736 tablet_bootstrap.cc:492] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: No bootstrap required, opened a new log
I20260928 21:59:28.624569 10736 ts_tablet_manager.cc:1397] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:28.625968 10736 raft_consensus.cc:348] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.626118 10736 raft_consensus.cc:374] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:28.626142 10736 raft_consensus.cc:729] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Initialized, Role: FOLLOWER
I20260928 21:59:28.626257 10736 consensus_queue.cc:260] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.626497 10736 ts_tablet_manager.cc:1428] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:28.626603 10564 heartbeater.cc:503] Master 127.2.165.126:42903 was elected leader, sending a full tablet report...
W20260928 21:59:28.649091 10565 tablet.cc:2369] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:28.704633 10738 raft_consensus.cc:482] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:28.704833 10738 raft_consensus.cc:504] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.705147 10738 leader_election.cc:288] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66:45485), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:28.708076 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.705460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:42375 (local address 127.2.165.67:32953)
0928 21:59:28.705524 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.705526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.705537 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:28.705563 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.705564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.705573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.705647 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:28.705806 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.706300 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.707301 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.707691 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.707789 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.707800 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.707968 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.708005 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:59:28.708096 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.705355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:32953 (local address 127.2.165.65:42375)
0928 21:59:28.705418 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.705432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.705495 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.705651 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.705652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.705653 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:28.705733 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.705735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.706391 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.706392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.707203 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.707205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.707789 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.707790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.707883 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.707886 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.707949 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.708002 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":28}
I20260928 21:59:28.708200 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.705478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:56423 (local address 127.2.165.66:45485)
0928 21:59:28.705525 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.705527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.705704 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:59:28.705735 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.705736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.705745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.705819 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:28.705977 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.706434 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.707471 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.707843 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.707932 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.707940 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.708095 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.708131 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:59:28.708352 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.705355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:45485 (local address 127.2.165.65:56423)
0928 21:59:28.705581 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:28.705597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:28.705674 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:28.705823 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:28.705824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:28.705826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:28.705896 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.705898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.706524 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.706525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.707371 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:28.707373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.707933 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.707934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:28.708014 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.708022 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:59:28.708090 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:59:28.708128 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":84,"thread_start_us":79,"threads_started":1}
I20260928 21:59:28.708453 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f76082bce41ba8020cc0cb2ad4d73" candidate_uuid: "65add314088c466ea3aaf1634e73c149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f" is_pre_election: true
I20260928 21:59:28.708624 10658 raft_consensus.cc:2454] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65add314088c466ea3aaf1634e73c149 in term 0.
I20260928 21:59:28.708663 10518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f76082bce41ba8020cc0cb2ad4d73" candidate_uuid: "65add314088c466ea3aaf1634e73c149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df9466c3691431ab1979d9f7d0f55ff" is_pre_election: true
I20260928 21:59:28.708833 10518 raft_consensus.cc:2454] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65add314088c466ea3aaf1634e73c149 in term 0.
I20260928 21:59:28.708938 10309 leader_election.cc:302] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370b233c35a84f86a4551b3306cd0c8f, 65add314088c466ea3aaf1634e73c149; no voters:
I20260928 21:59:28.709071 10738 raft_consensus.cc:2781] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:28.709156 10738 raft_consensus.cc:482] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:28.709192 10738 raft_consensus.cc:3037] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:28.710129 10738 raft_consensus.cc:504] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.710286 10738 leader_election.cc:288] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [CANDIDATE]: Term 1 election: Requested vote from peers 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66:45485), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:28.710443 10518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f76082bce41ba8020cc0cb2ad4d73" candidate_uuid: "65add314088c466ea3aaf1634e73c149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df9466c3691431ab1979d9f7d0f55ff"
I20260928 21:59:28.710489 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f76082bce41ba8020cc0cb2ad4d73" candidate_uuid: "65add314088c466ea3aaf1634e73c149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f"
I20260928 21:59:28.710563 10518 raft_consensus.cc:3037] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:28.710600 10658 raft_consensus.cc:3037] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:28.711295 10518 raft_consensus.cc:2454] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65add314088c466ea3aaf1634e73c149 in term 1.
I20260928 21:59:28.711375 10658 raft_consensus.cc:2454] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65add314088c466ea3aaf1634e73c149 in term 1.
I20260928 21:59:28.711463 10312 leader_election.cc:302] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65add314088c466ea3aaf1634e73c149, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:28.711570 10738 raft_consensus.cc:2781] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:28.711753 10738 raft_consensus.cc:686] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 1 LEADER]: Becoming Leader. State: Replica: 65add314088c466ea3aaf1634e73c149, State: Running, Role: LEADER
I20260928 21:59:28.711853 10738 consensus_queue.cc:237] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.712802 10220 catalog_manager.cc:5495] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65add314088c466ea3aaf1634e73c149 (127.2.165.65). New cstate: current_term: 1 leader_uuid: "65add314088c466ea3aaf1634e73c149" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:28.730017 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:28.727402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.2.165.65:37757)
0928 21:59:28.727478 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:28.727482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:28.727496 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:59:28.727532 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:28.727533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:28.727551 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:28.727618 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:28.727733 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.728208 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.729084 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:28.729287 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:28.729346 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:28.729609 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:28.729720 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:28.729809 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:28.729861 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:59:28.729881 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:59:28.731926 10658 raft_consensus.cc:1261] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Refusing update from remote peer 65add314088c466ea3aaf1634e73c149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:28.731926 10518 raft_consensus.cc:1261] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Refusing update from remote peer 65add314088c466ea3aaf1634e73c149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:28.732321 10738 consensus_queue.cc:1035] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:28.732537 10742 consensus_queue.cc:1035] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:59:28.732558 10425 tablet.cc:2369] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:28.735168 10747 mvcc.cc:205] Tried to move back new op lower bound from 7334431820723015680 to 7334431820644569088. Current Snapshot: MvccSnapshot[applied={T|T < 7334431820723015680}]
I20260928 21:59:28.735599 10746 mvcc.cc:205] Tried to move back new op lower bound from 7334431820723015680 to 7334431820644569088. Current Snapshot: MvccSnapshot[applied={T|T < 7334431820723015680}]
I20260928 21:59:28.737605 10220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:28.737723 10220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:28.738621 10220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Sending DeleteTablet for 3 replicas of tablet 529f76082bce41ba8020cc0cb2ad4d73
I20260928 21:59:28.738977 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet 529f76082bce41ba8020cc0cb2ad4d73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:28 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35790
I20260928 21:59:28.739003 10638 tablet_service.cc:1514] Processing DeleteTablet for tablet 529f76082bce41ba8020cc0cb2ad4d73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:28 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35902
I20260928 21:59:28.739367 10497 tablet_service.cc:1514] Processing DeleteTablet for tablet 529f76082bce41ba8020cc0cb2ad4d73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:28 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:47246
I20260928 21:59:28.739590 10756 tablet_replica.cc:330] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: stopping tablet replica
I20260928 21:59:28.739630 10755 tablet_replica.cc:330] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: stopping tablet replica
I20260928 21:59:28.739626 10220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:59:28.739738 10755 raft_consensus.cc:2229] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 1 LEADER]: Raft consensus shutting down.
W20260928 21:59:28.739777 10220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:28.739857 10755 raft_consensus.cc:2258] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:28.739979 10756 raft_consensus.cc:2229] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:28.740314 10756 raft_consensus.cc:2258] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:28.740832 10756 ts_tablet_manager.cc:1905] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:28.741027 10754 tablet_replica.cc:330] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: stopping tablet replica
I20260928 21:59:28.741343 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 8b3f313563c243fbb289c1925c611655 (DEFAULT_TABLE table=test-table [id=40f9e5c135214a86a53a26f23a9ecb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:28.741479 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b3f313563c243fbb289c1925c611655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:28.741494 10638 tablet_service.cc:1467] Processing CreateTablet for tablet 8b3f313563c243fbb289c1925c611655 (DEFAULT_TABLE table=test-table [id=40f9e5c135214a86a53a26f23a9ecb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:28.741695 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b3f313563c243fbb289c1925c611655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:28.742695 10756 ts_tablet_manager.cc:1918] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:28.742784 10756 log.cc:1196] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/529f76082bce41ba8020cc0cb2ad4d73
I20260928 21:59:28.742967 10733 tablet_bootstrap.cc:492] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: Bootstrap starting.
I20260928 21:59:28.743029 10497 tablet_service.cc:1467] Processing CreateTablet for tablet 8b3f313563c243fbb289c1925c611655 (DEFAULT_TABLE table=test-table [id=40f9e5c135214a86a53a26f23a9ecb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:28.743070 10732 tablet_bootstrap.cc:492] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Bootstrap starting.
I20260928 21:59:28.743125 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b3f313563c243fbb289c1925c611655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:28.743345 10754 raft_consensus.cc:2229] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:28.743405 10733 tablet_bootstrap.cc:654] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:28.743534 10732 tablet_bootstrap.cc:654] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:28.743558 10754 raft_consensus.cc:2258] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:28.743954 10756 ts_tablet_manager.cc:1939] T 529f76082bce41ba8020cc0cb2ad4d73 P 9df9466c3691431ab1979d9f7d0f55ff: Deleting consensus metadata
I20260928 21:59:28.744138 10754 ts_tablet_manager.cc:1905] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:28.744385 10736 tablet_bootstrap.cc:492] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: Bootstrap starting.
I20260928 21:59:28.744385 10208 catalog_manager.cc:4841] TS 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66:45485): tablet 529f76082bce41ba8020cc0cb2ad4d73 (table test-table [id=33f57b32c6994cd8a3a4690e17afd8e5]) successfully deleted
I20260928 21:59:28.744823 10736 tablet_bootstrap.cc:654] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:28.744844 10733 tablet_bootstrap.cc:492] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: No bootstrap required, opened a new log
I20260928 21:59:28.744884 10733 ts_tablet_manager.cc:1397] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:59:28.745043 10733 raft_consensus.cc:348] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.745090 10755 ts_tablet_manager.cc:1905] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:28.745126 10733 raft_consensus.cc:374] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:28.745149 10733 raft_consensus.cc:729] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65add314088c466ea3aaf1634e73c149, State: Initialized, Role: FOLLOWER
I20260928 21:59:28.745196 10733 consensus_queue.cc:260] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.745289 10733 ts_tablet_manager.cc:1428] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:28.745828 10736 tablet_bootstrap.cc:492] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: No bootstrap required, opened a new log
I20260928 21:59:28.745884 10736 ts_tablet_manager.cc:1397] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:28.746026 10736 raft_consensus.cc:348] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.746124 10736 raft_consensus.cc:374] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:28.746143 10736 raft_consensus.cc:729] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Initialized, Role: FOLLOWER
I20260928 21:59:28.746213 10736 consensus_queue.cc:260] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.746307 10736 ts_tablet_manager.cc:1428] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:28.746294 10754 ts_tablet_manager.cc:1918] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:28.746362 10754 log.cc:1196] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/529f76082bce41ba8020cc0cb2ad4d73
I20260928 21:59:28.746632 10754 ts_tablet_manager.cc:1939] T 529f76082bce41ba8020cc0cb2ad4d73 P 370b233c35a84f86a4551b3306cd0c8f: Deleting consensus metadata
I20260928 21:59:28.747072 10205 catalog_manager.cc:4841] TS 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953): tablet 529f76082bce41ba8020cc0cb2ad4d73 (table test-table [id=33f57b32c6994cd8a3a4690e17afd8e5]) successfully deleted
I20260928 21:59:28.747669 10755 ts_tablet_manager.cc:1918] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:28.747736 10755 log.cc:1196] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/529f76082bce41ba8020cc0cb2ad4d73
I20260928 21:59:28.747983 10755 ts_tablet_manager.cc:1939] T 529f76082bce41ba8020cc0cb2ad4d73 P 65add314088c466ea3aaf1634e73c149: Deleting consensus metadata
I20260928 21:59:28.748452 10205 catalog_manager.cc:4841] TS 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757): tablet 529f76082bce41ba8020cc0cb2ad4d73 (table test-table [id=33f57b32c6994cd8a3a4690e17afd8e5]) successfully deleted
I20260928 21:59:28.749097 10732 tablet_bootstrap.cc:492] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: No bootstrap required, opened a new log
I20260928 21:59:28.749166 10732 ts_tablet_manager.cc:1397] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 21:59:28.749343 10732 raft_consensus.cc:348] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.749432 10732 raft_consensus.cc:374] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:28.749468 10732 raft_consensus.cc:729] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370b233c35a84f86a4551b3306cd0c8f, State: Initialized, Role: FOLLOWER
I20260928 21:59:28.749521 10732 consensus_queue.cc:260] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:28.749639 10732 ts_tablet_manager.cc:1428] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260928 21:59:28.814188 10705 tablet.cc:2369] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:29.084424 10740 raft_consensus.cc:482] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:29.084534 10740 raft_consensus.cc:504] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.084977 10740 leader_election.cc:288] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.088250 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.085436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:46801 (local address 127.2.165.67:32953)
0928 21:59:29.085486 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.085488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.085497 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:29.085570 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.085571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.085581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.085632 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:29.085788 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.086287 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.087470 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.087871 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.087959 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.087969 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.088135 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.088172 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:59:29.088260 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.085253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:49481 (local address 127.2.165.65:37757)
0928 21:59:29.085329 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.085331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.085433 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:59:29.085614 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.085615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.085627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.085696 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:29.085859 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.086332 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.087461 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.087892 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.087971 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.087998 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.088146 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.088188 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:59:29.088274 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.085346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:32953 (local address 127.2.165.66:46801)
0928 21:59:29.085422 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:29.085433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:29.085489 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:29.085634 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:29.085635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:29.085637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:29.085712 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.085713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.086396 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.086397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.087358 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.087361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.087958 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.087959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.088053 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.088063 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:59:29.088129 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:59:29.088184 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":2,"thread_start_us":38,"threads_started":1}
I20260928 21:59:29.088279 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.085171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:37757 (local address 127.2.165.66:49481)
0928 21:59:29.085322 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:29.085336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:29.085444 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:29.085700 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:29.085701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:29.085703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:29.085766 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.085768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.086417 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.086418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.087366 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.087368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.087980 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.087981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.088063 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.088067 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:29.088127 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:59:29.088189 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":20,"thread_start_us":71,"threads_started":1}
I20260928 21:59:29.088568 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3f313563c243fbb289c1925c611655" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f" is_pre_election: true
I20260928 21:59:29.088699 10658 raft_consensus.cc:2454] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.088891 10449 leader_election.cc:302] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370b233c35a84f86a4551b3306cd0c8f, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.088893 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3f313563c243fbb289c1925c611655" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149" is_pre_election: true
I20260928 21:59:29.089030 10378 raft_consensus.cc:2454] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.089039 10740 raft_consensus.cc:2781] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:29.089076 10740 raft_consensus.cc:482] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:29.089095 10740 raft_consensus.cc:3037] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.089825 10740 raft_consensus.cc:504] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.089967 10740 leader_election.cc:288] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Requested vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.090126 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3f313563c243fbb289c1925c611655" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149"
I20260928 21:59:29.090153 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b3f313563c243fbb289c1925c611655" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f"
I20260928 21:59:29.090226 10378 raft_consensus.cc:3037] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.090261 10658 raft_consensus.cc:3037] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.090895 10378 raft_consensus.cc:2454] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.090935 10658 raft_consensus.cc:2454] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.091104 10449 leader_election.cc:302] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65add314088c466ea3aaf1634e73c149, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.091231 10740 raft_consensus.cc:2781] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:29.091372 10740 raft_consensus.cc:686] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 LEADER]: Becoming Leader. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Running, Role: LEADER
I20260928 21:59:29.091449 10740 consensus_queue.cc:237] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.092183 10220 catalog_manager.cc:5495] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66). New cstate: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632767, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.126@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:29.129544 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.121439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:42903 (local address 127.0.0.1:42562)
0928 21:59:29.121612 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:29.121625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:29.121711 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:29.121935 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:29.121936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:29.122265 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:29.122313 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.122315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.123014 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.123015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.123623 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.123625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.124006 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.124006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.124050 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.124646 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:29.124650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:29.126425 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:29.128211 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:29.128219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:29.128221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:29.128430 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:29.128894 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:29.128895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:29.128896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:29.128960 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:29.129261 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:29.129268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:29.129311 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:59:29.129365 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:59:29.129467 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260928 21:59:29.129666 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.121555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.2.165.126:42903)
0928 21:59:29.121779 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.121781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.121791 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:29.121813 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.121814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.121825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.121894 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:29.122412 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.122938 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.123721 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.123933 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.123997 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.126533 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:29.126534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:29.126534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:29.126578 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:29.128101 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:29.128591 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:29.128593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:29.128593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:29.128656 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:29.129051 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:29.129052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:29.129053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:29.129187 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:29.129269 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.129409 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.129522 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":204,"thread_start_us":162,"threads_started":1}
I20260928 21:59:29.131630 10220 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:42562:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:59:29.131880 10220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:29.133526 10638 tablet_service.cc:1467] Processing CreateTablet for tablet aaa5e2e3ac704ba2822f401ca3b74e56 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0bd2a63e4af415f8579a9d5fb8d9c49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:59:29.133502 10358 tablet_service.cc:1467] Processing CreateTablet for tablet aaa5e2e3ac704ba2822f401ca3b74e56 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0bd2a63e4af415f8579a9d5fb8d9c49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:59:29.133523 10497 tablet_service.cc:1467] Processing CreateTablet for tablet aaa5e2e3ac704ba2822f401ca3b74e56 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0bd2a63e4af415f8579a9d5fb8d9c49]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:59:29.133665 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa5e2e3ac704ba2822f401ca3b74e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.133678 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa5e2e3ac704ba2822f401ca3b74e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.133700 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa5e2e3ac704ba2822f401ca3b74e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.134989 10736 tablet_bootstrap.cc:492] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: Bootstrap starting.
I20260928 21:59:29.135162 10733 tablet_bootstrap.cc:492] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: Bootstrap starting.
I20260928 21:59:29.135178 10732 tablet_bootstrap.cc:492] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: Bootstrap starting.
I20260928 21:59:29.135456 10736 tablet_bootstrap.cc:654] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.135504 10732 tablet_bootstrap.cc:654] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.135499 10733 tablet_bootstrap.cc:654] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.136107 10732 tablet_bootstrap.cc:492] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: No bootstrap required, opened a new log
I20260928 21:59:29.136181 10736 tablet_bootstrap.cc:492] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: No bootstrap required, opened a new log
I20260928 21:59:29.136194 10732 ts_tablet_manager.cc:1397] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:29.136247 10736 ts_tablet_manager.cc:1397] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:29.136304 10733 tablet_bootstrap.cc:492] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: No bootstrap required, opened a new log
I20260928 21:59:29.136361 10733 ts_tablet_manager.cc:1397] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:29.136373 10732 raft_consensus.cc:348] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.136457 10732 raft_consensus.cc:374] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.136420 10736 raft_consensus.cc:348] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.136484 10732 raft_consensus.cc:729] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370b233c35a84f86a4551b3306cd0c8f, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.136500 10736 raft_consensus.cc:374] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.136466 10733 raft_consensus.cc:348] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.136523 10736 raft_consensus.cc:729] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.136529 10733 raft_consensus.cc:374] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.136528 10732 consensus_queue.cc:260] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.136556 10733 raft_consensus.cc:729] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65add314088c466ea3aaf1634e73c149, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.136575 10736 consensus_queue.cc:260] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.136595 10733 consensus_queue.cc:260] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.136629 10732 ts_tablet_manager.cc:1428] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:29.136721 10733 ts_tablet_manager.cc:1428] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:29.136749 10736 ts_tablet_manager.cc:1428] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:29.136883 10740 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.136879 10737 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.136966 10740 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:59:29.136967 10737 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:59:29.137069 10742 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.137167 10742 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:59:29.499437 10740 raft_consensus.cc:482] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:29.499560 10740 raft_consensus.cc:504] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.499747 10740 leader_election.cc:288] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.500097 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa5e2e3ac704ba2822f401ca3b74e56" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f" is_pre_election: true
I20260928 21:59:29.500099 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa5e2e3ac704ba2822f401ca3b74e56" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149" is_pre_election: true
I20260928 21:59:29.500244 10658 raft_consensus.cc:2454] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.500244 10378 raft_consensus.cc:2454] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.500408 10449 leader_election.cc:302] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370b233c35a84f86a4551b3306cd0c8f, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.500523 10740 raft_consensus.cc:2781] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:29.500571 10740 raft_consensus.cc:482] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:29.500597 10740 raft_consensus.cc:3037] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.501392 10740 raft_consensus.cc:504] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.501539 10740 leader_election.cc:288] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Requested vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.501770 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa5e2e3ac704ba2822f401ca3b74e56" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149"
I20260928 21:59:29.501888 10378 raft_consensus.cc:3037] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.501814 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa5e2e3ac704ba2822f401ca3b74e56" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f"
I20260928 21:59:29.501940 10658 raft_consensus.cc:3037] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.502547 10378 raft_consensus.cc:2454] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.502553 10658 raft_consensus.cc:2454] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.502727 10449 leader_election.cc:302] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370b233c35a84f86a4551b3306cd0c8f, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.502869 10740 raft_consensus.cc:2781] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:29.502926 10740 raft_consensus.cc:686] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 LEADER]: Becoming Leader. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Running, Role: LEADER
I20260928 21:59:29.503005 10740 consensus_queue.cc:237] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.503330 10762 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: TxnStatusTablet state changed. Reason: New leader 9df9466c3691431ab1979d9f7d0f55ff. Latest consensus state: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.503454 10762 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:59:29.503630 10775 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:59:29.503767 10775 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:59:29.503707 10220 catalog_manager.cc:5495] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66). New cstate: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:29.532828 10762 consensus_queue.cc:1035] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.534937 10740 consensus_queue.cc:1035] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.585098 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.582071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:45485 (local address 127.0.0.1:47250)
0928 21:59:29.582159 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:29.582172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:29.582232 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:29.582520 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:29.582521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:29.582522 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:59:29.582587 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.582591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.583433 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.583434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.584027 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.584029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.584447 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.584448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.584523 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.584541 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:59:29.584890 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:59:29.584900 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:59:29.584958 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:59:29.585016 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:59:29.585116 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.582175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.2.165.66:45485)
0928 21:59:29.582366 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.582369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.582375 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:59:29.582407 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.582407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.582425 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.582515 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:29.582830 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.583340 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.584133 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.584366 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.584447 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.584702 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:29.584817 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:29.584908 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.584967 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.584993 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260928 21:59:29.586730 10658 raft_consensus.cc:1261] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Refusing update from remote peer 9df9466c3691431ab1979d9f7d0f55ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:29.586737 10378 raft_consensus.cc:1261] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Refusing update from remote peer 9df9466c3691431ab1979d9f7d0f55ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:29.587020 10762 consensus_queue.cc:1035] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.587188 10740 consensus_queue.cc:1035] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.587723 10742 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: TxnStatusTablet state changed. Reason: New leader 9df9466c3691431ab1979d9f7d0f55ff. Latest consensus state: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.587739 10782 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: TxnStatusTablet state changed. Reason: New leader 9df9466c3691431ab1979d9f7d0f55ff. Latest consensus state: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.587816 10742 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:59:29.587839 10782 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:59:29.588804 10762 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.588901 10762 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:59:29.588948 10782 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.589011 10782 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 370b233c35a84f86a4551b3306cd0c8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:59:29.589105 10742 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.589192 10742 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 65add314088c466ea3aaf1634e73c149: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:59:29.588909 10740 tablet_replica.cc:439] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } } }
I20260928 21:59:29.589721 10740 tablet_replica.cc:442] T aaa5e2e3ac704ba2822f401ca3b74e56 P 9df9466c3691431ab1979d9f7d0f55ff: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:59:29.589995 10779 mvcc.cc:205] Tried to move back new op lower bound from 7334431824225300480 to 7334431823884931072. Current Snapshot: MvccSnapshot[applied={T|T < 7334431824225300480}]
I20260928 21:59:29.594002 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.590951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.2.165.66:45485)
0928 21:59:29.590995 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.590997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.591062 (+ 65us) server_negotiation.cc:407] Connection header received
0928 21:59:29.591170 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.591171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.591180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.591227 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:29.591382 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.591896 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.593087 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.593279 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.593400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.593559 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:29.593632 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:29.593684 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.593817 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.593890 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:59:29.599566 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.597085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:50037 (local address 127.2.165.66:45485)
0928 21:59:29.597158 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.597160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.597227 (+ 67us) server_negotiation.cc:407] Connection header received
0928 21:59:29.597300 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.597301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.597312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.597380 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:29.597488 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.597986 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.598764 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.599044 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.599116 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.599295 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:29.599327 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:29.599400 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.599457 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.599478 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260928 21:59:29.599566 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.597010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:45485 (local address 127.2.165.66:50037)
0928 21:59:29.597166 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:29.597177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:29.597243 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:29.597381 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:29.597382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:29.597383 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:59:29.597440 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.597441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.598079 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.598081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.598665 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:29.598668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.599113 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.599114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:29.599167 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.599183 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:59:29.599399 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:59:29.599400 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:59:29.599452 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:59:29.599500 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260928 21:59:29.615478 10220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:29.615612 10220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:29.616326 10220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Sending DeleteTablet for 3 replicas of tablet 8b3f313563c243fbb289c1925c611655
I20260928 21:59:29.616726 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b3f313563c243fbb289c1925c611655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35790
I20260928 21:59:29.616818 10638 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b3f313563c243fbb289c1925c611655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35902
I20260928 21:59:29.616873 10497 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b3f313563c243fbb289c1925c611655 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:47246
I20260928 21:59:29.617084 10797 tablet_replica.cc:330] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: stopping tablet replica
I20260928 21:59:29.617094 10799 tablet_replica.cc:330] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: stopping tablet replica
I20260928 21:59:29.617159 10799 raft_consensus.cc:2229] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:29.617143 10797 raft_consensus.cc:2229] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:29.617216 10797 raft_consensus.cc:2258] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.617251 10799 raft_consensus.cc:2258] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.617492 10797 ts_tablet_manager.cc:1905] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.617506 10799 ts_tablet_manager.cc:1905] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.619310 10797 ts_tablet_manager.cc:1918] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:29.619366 10797 log.cc:1196] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/8b3f313563c243fbb289c1925c611655
I20260928 21:59:29.619511 10799 ts_tablet_manager.cc:1918] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:29.619568 10799 log.cc:1196] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/8b3f313563c243fbb289c1925c611655
I20260928 21:59:29.619596 10797 ts_tablet_manager.cc:1939] T 8b3f313563c243fbb289c1925c611655 P 65add314088c466ea3aaf1634e73c149: Deleting consensus metadata
I20260928 21:59:29.619825 10799 ts_tablet_manager.cc:1939] T 8b3f313563c243fbb289c1925c611655 P 9df9466c3691431ab1979d9f7d0f55ff: Deleting consensus metadata
I20260928 21:59:29.620117 10205 catalog_manager.cc:4841] TS 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757): tablet 8b3f313563c243fbb289c1925c611655 (table test-table [id=40f9e5c135214a86a53a26f23a9ecb9b]) successfully deleted
I20260928 21:59:29.620436 10208 catalog_manager.cc:4841] TS 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66:45485): tablet 8b3f313563c243fbb289c1925c611655 (table test-table [id=40f9e5c135214a86a53a26f23a9ecb9b]) successfully deleted
I20260928 21:59:29.620802 10798 tablet_replica.cc:330] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: stopping tablet replica
I20260928 21:59:29.620885 10798 raft_consensus.cc:2229] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:29.620935 10798 raft_consensus.cc:2258] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.621213 10798 ts_tablet_manager.cc:1905] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.622870 10798 ts_tablet_manager.cc:1918] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:29.622936 10798 log.cc:1196] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/8b3f313563c243fbb289c1925c611655
I20260928 21:59:29.623150 10798 ts_tablet_manager.cc:1939] T 8b3f313563c243fbb289c1925c611655 P 370b233c35a84f86a4551b3306cd0c8f: Deleting consensus metadata
I20260928 21:59:29.623626 10205 catalog_manager.cc:4841] TS 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953): tablet 8b3f313563c243fbb289c1925c611655 (table test-table [id=40f9e5c135214a86a53a26f23a9ecb9b]) successfully deleted
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:59:29.645241 10807 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:29.639398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.2.165.126:42903)
0928 21:59:29.639582 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.639585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.639597 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:59:29.639672 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.639673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.639685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.639750 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:29.640455 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.641114 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.641966 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.642175 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.642262 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.645126 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42568: BlockingRecv error: recv got EOF from 127.0.0.1:42568 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM
I20260928 21:59:29.653654 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.646054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.2.165.126:42903)
0928 21:59:29.646101 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.646103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.646183 (+ 80us) server_negotiation.cc:407] Connection header received
0928 21:59:29.646240 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.646241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.646253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.646304 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:29.646740 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.647507 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.648175 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.648483 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.648547 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.650912 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:29.650913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:29.650913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:29.650921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:29.652416 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:29.652790 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:29.652791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:29.652792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:29.652855 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:29.653036 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:29.653037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:29.653038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:29.653178 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:29.653298 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.653475 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.653563 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:59:29.655359 10220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:29.655546 10220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:29.656903 10497 tablet_service.cc:1467] Processing CreateTablet for tablet b9c54f403bbd4f8abdaba039fd9cbb2e (DEFAULT_TABLE table=test-table [id=2efc166011014af08bd639fe89694963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:29.656924 10358 tablet_service.cc:1467] Processing CreateTablet for tablet b9c54f403bbd4f8abdaba039fd9cbb2e (DEFAULT_TABLE table=test-table [id=2efc166011014af08bd639fe89694963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:29.656955 10638 tablet_service.cc:1467] Processing CreateTablet for tablet b9c54f403bbd4f8abdaba039fd9cbb2e (DEFAULT_TABLE table=test-table [id=2efc166011014af08bd639fe89694963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:29.657047 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9c54f403bbd4f8abdaba039fd9cbb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.657058 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9c54f403bbd4f8abdaba039fd9cbb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.657058 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9c54f403bbd4f8abdaba039fd9cbb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.658660 10815 tablet_bootstrap.cc:492] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: Bootstrap starting.
I20260928 21:59:29.658805 10814 tablet_bootstrap.cc:492] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: Bootstrap starting.
I20260928 21:59:29.659126 10815 tablet_bootstrap.cc:654] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.659198 10814 tablet_bootstrap.cc:654] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.659494 10813 tablet_bootstrap.cc:492] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: Bootstrap starting.
I20260928 21:59:29.659862 10813 tablet_bootstrap.cc:654] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.659996 10814 tablet_bootstrap.cc:492] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: No bootstrap required, opened a new log
I20260928 21:59:29.660053 10814 ts_tablet_manager.cc:1397] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:29.660194 10814 raft_consensus.cc:348] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.660250 10815 tablet_bootstrap.cc:492] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: No bootstrap required, opened a new log
I20260928 21:59:29.660280 10814 raft_consensus.cc:374] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.660292 10815 ts_tablet_manager.cc:1397] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:29.660313 10814 raft_consensus.cc:729] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65add314088c466ea3aaf1634e73c149, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.660383 10814 consensus_queue.cc:260] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.660444 10815 raft_consensus.cc:348] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.660517 10815 raft_consensus.cc:374] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.660542 10815 raft_consensus.cc:729] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370b233c35a84f86a4551b3306cd0c8f, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.660607 10815 consensus_queue.cc:260] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.660745 10815 ts_tablet_manager.cc:1428] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:29.660964 10813 tablet_bootstrap.cc:492] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: No bootstrap required, opened a new log
I20260928 21:59:29.661003 10813 ts_tablet_manager.cc:1397] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:29.661121 10813 raft_consensus.cc:348] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.661197 10813 raft_consensus.cc:374] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.661219 10813 raft_consensus.cc:729] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.661283 10813 consensus_queue.cc:260] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.661386 10813 ts_tablet_manager.cc:1428] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:29.661626 10814 ts_tablet_manager.cc:1428] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:29.789017 10740 raft_consensus.cc:482] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:29.789124 10740 raft_consensus.cc:504] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.789328 10740 leader_election.cc:288] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.789631 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c54f403bbd4f8abdaba039fd9cbb2e" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149" is_pre_election: true
I20260928 21:59:29.789741 10378 raft_consensus.cc:2454] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.789701 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c54f403bbd4f8abdaba039fd9cbb2e" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f" is_pre_election: true
I20260928 21:59:29.789857 10658 raft_consensus.cc:2454] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.789958 10449 leader_election.cc:302] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65add314088c466ea3aaf1634e73c149, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.790088 10740 raft_consensus.cc:2781] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:29.790131 10740 raft_consensus.cc:482] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:29.790153 10740 raft_consensus.cc:3037] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.790822 10740 raft_consensus.cc:504] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.790968 10740 leader_election.cc:288] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Requested vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.791128 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c54f403bbd4f8abdaba039fd9cbb2e" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149"
I20260928 21:59:29.791153 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c54f403bbd4f8abdaba039fd9cbb2e" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f"
I20260928 21:59:29.791214 10658 raft_consensus.cc:3037] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.791210 10378 raft_consensus.cc:3037] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.791846 10658 raft_consensus.cc:2454] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.791875 10378 raft_consensus.cc:2454] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.792042 10449 leader_election.cc:302] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65add314088c466ea3aaf1634e73c149, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.792151 10740 raft_consensus.cc:2781] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:29.792204 10740 raft_consensus.cc:686] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 1 LEADER]: Becoming Leader. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Running, Role: LEADER
I20260928 21:59:29.792250 10740 consensus_queue.cc:237] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.792950 10221 catalog_manager.cc:5495] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66). New cstate: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:29.808007 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.805693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.2.165.66:45485)
0928 21:59:29.805803 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:29.805804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:29.805811 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:59:29.805832 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:29.805833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:29.805842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:29.805904 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:29.806060 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.806539 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.807328 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:29.807521 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:29.807597 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:29.807735 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:29.807770 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:29.807831 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:29.807893 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:59:29.807911 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 21:59:29.808946 10378 raft_consensus.cc:1261] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Refusing update from remote peer 9df9466c3691431ab1979d9f7d0f55ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:29.808946 10658 raft_consensus.cc:1261] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Refusing update from remote peer 9df9466c3691431ab1979d9f7d0f55ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:29.809267 10762 consensus_queue.cc:1035] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.809383 10740 consensus_queue.cc:1035] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.811945 10220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:29.812040 10220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:29.812623 10220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Sending DeleteTablet for 3 replicas of tablet b9c54f403bbd4f8abdaba039fd9cbb2e
I20260928 21:59:29.812923 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c54f403bbd4f8abdaba039fd9cbb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35790
I20260928 21:59:29.812935 10497 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c54f403bbd4f8abdaba039fd9cbb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:47246
I20260928 21:59:29.813038 10799 tablet_replica.cc:330] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: stopping tablet replica
I20260928 21:59:29.813027 10638 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c54f403bbd4f8abdaba039fd9cbb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35902
I20260928 21:59:29.813097 10799 raft_consensus.cc:2229] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:29.813105 10797 tablet_replica.cc:330] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: stopping tablet replica
I20260928 21:59:29.813154 10797 raft_consensus.cc:2229] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:29.813169 10799 raft_consensus.cc:2258] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.813211 10797 raft_consensus.cc:2258] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.813288 10798 tablet_replica.cc:330] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: stopping tablet replica
I20260928 21:59:29.813346 10798 raft_consensus.cc:2229] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:29.813390 10798 raft_consensus.cc:2258] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.813364 10220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:59:29.813493 10799 ts_tablet_manager.cc:1905] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:59:29.813505 10220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:29.813733 10797 ts_tablet_manager.cc:1905] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.813771 10798 ts_tablet_manager.cc:1905] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.815241 10799 ts_tablet_manager.cc:1918] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:29.815311 10799 log.cc:1196] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/b9c54f403bbd4f8abdaba039fd9cbb2e
I20260928 21:59:29.815310 10798 ts_tablet_manager.cc:1918] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:29.815369 10798 log.cc:1196] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/b9c54f403bbd4f8abdaba039fd9cbb2e
I20260928 21:59:29.815394 10797 ts_tablet_manager.cc:1918] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:29.815454 10797 log.cc:1196] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/b9c54f403bbd4f8abdaba039fd9cbb2e
I20260928 21:59:29.815543 10799 ts_tablet_manager.cc:1939] T b9c54f403bbd4f8abdaba039fd9cbb2e P 9df9466c3691431ab1979d9f7d0f55ff: Deleting consensus metadata
I20260928 21:59:29.815595 10798 ts_tablet_manager.cc:1939] T b9c54f403bbd4f8abdaba039fd9cbb2e P 370b233c35a84f86a4551b3306cd0c8f: Deleting consensus metadata
I20260928 21:59:29.815693 10797 ts_tablet_manager.cc:1939] T b9c54f403bbd4f8abdaba039fd9cbb2e P 65add314088c466ea3aaf1634e73c149: Deleting consensus metadata
I20260928 21:59:29.815917 10205 catalog_manager.cc:4841] TS 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953): tablet b9c54f403bbd4f8abdaba039fd9cbb2e (table test-table [id=2efc166011014af08bd639fe89694963]) successfully deleted
I20260928 21:59:29.815898 10208 catalog_manager.cc:4841] TS 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66:45485): tablet b9c54f403bbd4f8abdaba039fd9cbb2e (table test-table [id=2efc166011014af08bd639fe89694963]) successfully deleted
I20260928 21:59:29.816154 10205 catalog_manager.cc:4841] TS 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757): tablet b9c54f403bbd4f8abdaba039fd9cbb2e (table test-table [id=2efc166011014af08bd639fe89694963]) successfully deleted
I20260928 21:59:29.817016 10638 tablet_service.cc:1467] Processing CreateTablet for tablet 9c99eea457e44340b073d5de6f3eb54a (DEFAULT_TABLE table=test-table [id=01becc6f77bf4e6fa7ba4448815de90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:29.817148 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c99eea457e44340b073d5de6f3eb54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.817180 10497 tablet_service.cc:1467] Processing CreateTablet for tablet 9c99eea457e44340b073d5de6f3eb54a (DEFAULT_TABLE table=test-table [id=01becc6f77bf4e6fa7ba4448815de90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:29.817274 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c99eea457e44340b073d5de6f3eb54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.818449 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 9c99eea457e44340b073d5de6f3eb54a (DEFAULT_TABLE table=test-table [id=01becc6f77bf4e6fa7ba4448815de90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:29.818509 10813 tablet_bootstrap.cc:492] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: Bootstrap starting.
I20260928 21:59:29.818573 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c99eea457e44340b073d5de6f3eb54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:29.818650 10815 tablet_bootstrap.cc:492] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: Bootstrap starting.
I20260928 21:59:29.818864 10813 tablet_bootstrap.cc:654] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.819366 10815 tablet_bootstrap.cc:654] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.820048 10815 tablet_bootstrap.cc:492] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: No bootstrap required, opened a new log
I20260928 21:59:29.820060 10814 tablet_bootstrap.cc:492] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149: Bootstrap starting.
I20260928 21:59:29.820101 10815 ts_tablet_manager.cc:1397] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:29.820257 10815 raft_consensus.cc:348] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.820335 10815 raft_consensus.cc:374] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.820362 10815 raft_consensus.cc:729] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370b233c35a84f86a4551b3306cd0c8f, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.820425 10814 tablet_bootstrap.cc:654] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:29.820426 10815 consensus_queue.cc:260] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.820536 10815 ts_tablet_manager.cc:1428] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:29.820832 10813 tablet_bootstrap.cc:492] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: No bootstrap required, opened a new log
I20260928 21:59:29.820894 10813 ts_tablet_manager.cc:1397] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:59:29.820964 10814 tablet_bootstrap.cc:492] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149: No bootstrap required, opened a new log
I20260928 21:59:29.821012 10814 ts_tablet_manager.cc:1397] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:29.821009 10813 raft_consensus.cc:348] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.821087 10813 raft_consensus.cc:374] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.821112 10813 raft_consensus.cc:729] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.821130 10814 raft_consensus.cc:348] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.821172 10813 consensus_queue.cc:260] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.821214 10814 raft_consensus.cc:374] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:29.821240 10814 raft_consensus.cc:729] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65add314088c466ea3aaf1634e73c149, State: Initialized, Role: FOLLOWER
I20260928 21:59:29.821270 10813 ts_tablet_manager.cc:1428] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:29.821297 10814 consensus_queue.cc:260] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.821412 10814 ts_tablet_manager.cc:1428] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:59:29.848474 10740 raft_consensus.cc:482] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:29.848606 10740 raft_consensus.cc:504] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.848863 10740 leader_election.cc:288] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.849098 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c99eea457e44340b073d5de6f3eb54a" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f" is_pre_election: true
I20260928 21:59:29.849092 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c99eea457e44340b073d5de6f3eb54a" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149" is_pre_election: true
I20260928 21:59:29.849188 10658 raft_consensus.cc:2454] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.849237 10378 raft_consensus.cc:2454] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 0.
I20260928 21:59:29.849377 10449 leader_election.cc:302] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370b233c35a84f86a4551b3306cd0c8f, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.849481 10740 raft_consensus.cc:2781] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:29.849548 10740 raft_consensus.cc:482] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:29.849577 10740 raft_consensus.cc:3037] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.850253 10740 raft_consensus.cc:504] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.850399 10740 leader_election.cc:288] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Requested vote from peers 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757), 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953)
I20260928 21:59:29.850586 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c99eea457e44340b073d5de6f3eb54a" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65add314088c466ea3aaf1634e73c149"
I20260928 21:59:29.850680 10378 raft_consensus.cc:3037] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.850680 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c99eea457e44340b073d5de6f3eb54a" candidate_uuid: "9df9466c3691431ab1979d9f7d0f55ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370b233c35a84f86a4551b3306cd0c8f"
I20260928 21:59:29.850780 10658 raft_consensus.cc:3037] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:29.851297 10378 raft_consensus.cc:2454] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.851513 10449 leader_election.cc:302] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65add314088c466ea3aaf1634e73c149, 9df9466c3691431ab1979d9f7d0f55ff; no voters:
I20260928 21:59:29.851505 10658 raft_consensus.cc:2454] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df9466c3691431ab1979d9f7d0f55ff in term 1.
I20260928 21:59:29.851665 10740 raft_consensus.cc:2781] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:29.851743 10740 raft_consensus.cc:686] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 1 LEADER]: Becoming Leader. State: Replica: 9df9466c3691431ab1979d9f7d0f55ff, State: Running, Role: LEADER
I20260928 21:59:29.851804 10740 consensus_queue.cc:237] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } }
I20260928 21:59:29.852484 10220 catalog_manager.cc:5495] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66). New cstate: current_term: 1 leader_uuid: "9df9466c3691431ab1979d9f7d0f55ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df9466c3691431ab1979d9f7d0f55ff" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:29.859948 10658 raft_consensus.cc:1261] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Refusing update from remote peer 9df9466c3691431ab1979d9f7d0f55ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:29.859958 10378 raft_consensus.cc:1261] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Refusing update from remote peer 9df9466c3691431ab1979d9f7d0f55ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:29.860167 10762 consensus_queue.cc:1035] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "65add314088c466ea3aaf1634e73c149" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.860298 10787 consensus_queue.cc:1035] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "370b233c35a84f86a4551b3306cd0c8f" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:29.870549 10221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:29.870657 10221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:29.871299 10221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1cdb59c72c4493d84e1f10b095a8077: Sending DeleteTablet for 3 replicas of tablet 9c99eea457e44340b073d5de6f3eb54a
I20260928 21:59:29.871548 10497 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c99eea457e44340b073d5de6f3eb54a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:47246
I20260928 21:59:29.871605 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c99eea457e44340b073d5de6f3eb54a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35790
I20260928 21:59:29.871603 10638 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c99eea457e44340b073d5de6f3eb54a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:29 UTC) from {username='oryx', principal='oryx/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35902
I20260928 21:59:29.871699 10798 tablet_replica.cc:330] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: stopping tablet replica
I20260928 21:59:29.871723 10799 tablet_replica.cc:330] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: stopping tablet replica
I20260928 21:59:29.871699 10797 tablet_replica.cc:330] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149: stopping tablet replica
I20260928 21:59:29.871753 10798 raft_consensus.cc:2229] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:29.871771 10799 raft_consensus.cc:2229] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:29.871788 10798 raft_consensus.cc:2258] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.871801 10797 raft_consensus.cc:2229] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:29.871838 10797 raft_consensus.cc:2258] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.871838 10799 raft_consensus.cc:2258] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:29.872212 10798 ts_tablet_manager.cc:1905] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.872220 10797 ts_tablet_manager.cc:1905] T 9c99eea457e44340b073d5de6f3eb54a P 65add314088c466ea3aaf1634e73c149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.872205 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 10293
I20260928 21:59:29.874166 10798 ts_tablet_manager.cc:1918] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:29.874235 10798 log.cc:1196] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/9c99eea457e44340b073d5de6f3eb54a
I20260928 21:59:29.874477 10798 ts_tablet_manager.cc:1939] T 9c99eea457e44340b073d5de6f3eb54a P 370b233c35a84f86a4551b3306cd0c8f: Deleting consensus metadata
I20260928 21:59:29.874897 10205 catalog_manager.cc:4841] TS 370b233c35a84f86a4551b3306cd0c8f (127.2.165.67:32953): tablet 9c99eea457e44340b073d5de6f3eb54a (table test-table [id=01becc6f77bf4e6fa7ba4448815de90a]) successfully deleted
I20260928 21:59:29.874913 10799 ts_tablet_manager.cc:1905] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:29.876797 10799 ts_tablet_manager.cc:1918] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:59:29.876863 10799 log.cc:1196] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/9c99eea457e44340b073d5de6f3eb54a
I20260928 21:59:29.877070 10799 ts_tablet_manager.cc:1939] T 9c99eea457e44340b073d5de6f3eb54a P 9df9466c3691431ab1979d9f7d0f55ff: Deleting consensus metadata
I20260928 21:59:29.877627 10208 catalog_manager.cc:4841] TS 9df9466c3691431ab1979d9f7d0f55ff (127.2.165.66:45485): tablet 9c99eea457e44340b073d5de6f3eb54a (table test-table [id=01becc6f77bf4e6fa7ba4448815de90a]) successfully deleted
W20260928 21:59:29.879925 10205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:37757 (error 108)
I20260928 21:59:29.880152 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 10433
I20260928 21:59:29.881832 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:29.881576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:37757 (local address 127.0.0.1:35804)
0928 21:59:29.881736 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:29.881792 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:37757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
W20260928 21:59:29.881953 10205 catalog_manager.cc:4584] TS 65add314088c466ea3aaf1634e73c149 (127.2.165.65:37757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c99eea457e44340b073d5de6f3eb54a: Network error: Client connection negotiation failed: client connection to 127.2.165.65:37757: connect: Connection refused (error 111)
I20260928 21:59:29.886387 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 10573
I20260928 21:59:29.891422 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 10189
2026-09-28T21:59:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3173 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10833](info): set up 2 sockets
Sep 28 21:59:29 dist-test-slave-3t36 krb5kdc[10833](info): commencing operation
krb5kdc: starting...
W20260928 21:59:30.977428 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:31 dist-test-slave-3t36 krb5kdc[10833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:31.105782 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38355
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:38355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:31.188145 10846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:31.188366 10846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:31.188388 10846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:31.189522 10846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:31.189563 10846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:31.189579 10846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:31.189591 10846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:31.189603 10846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:31.191063 10846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:38355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:31.191255 10846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:31.191426 10846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:31.196375 10851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:31.196408 10852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:31.196408 10854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:31.196519 10846 server_base.cc:1034] running on GCE node
I20260928 21:59:31.197196 10846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:31.197505 10846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:31.198683 10846 hybrid_clock.cc:648] HybridClock initialized: now 1790632771198661 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T21:59:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1310 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:31 dist-test-slave-3t36 krb5kdc[10859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:31 dist-test-slave-3t36 krb5kdc[10859](info): set up 2 sockets
Sep 28 21:59:31 dist-test-slave-3t36 krb5kdc[10859](info): commencing operation
krb5kdc: starting...
W20260928 21:59:32.238561 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:32.367942 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35057
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:35057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:32.458237 10872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:32.458489 10872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:32.458514 10872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:32.459897 10872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:32.459959 10872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:32.459975 10872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:32.459987 10872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:32.460000 10872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:32.461715 10872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:35057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:32.462044 10872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:32.462303 10872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:32.467519 10877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:32.467581 10880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:32.467811 10872 server_base.cc:1034] running on GCE node
W20260928 21:59:32.467581 10878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:32.468140 10872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:32.468463 10872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:32.469645 10872 hybrid_clock.cc:648] HybridClock initialized: now 1790632772469588 us; error 81 us; skew 500 ppm
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:32.472496 10872 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:32.473102 10872 webserver.cc:460] Webserver started at http://127.2.165.126:45797/ using document root <none> and password file <none>
I20260928 21:59:32.473357 10872 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:32.473443 10872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:32.473593 10872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:32.474814 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "7886fa055fe04da5817a68490f47c300"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "f059973683d5d01c3463228e1b4d66d2"
server_key_iv: "a5439d31b6055087b3641f3039ad8f00"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.475189 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "7886fa055fe04da5817a68490f47c300"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "f059973683d5d01c3463228e1b4d66d2"
server_key_iv: "a5439d31b6055087b3641f3039ad8f00"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.476613 10872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:32.477624 10887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.477860 10872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:32.477981 10872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "7886fa055fe04da5817a68490f47c300"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "f059973683d5d01c3463228e1b4d66d2"
server_key_iv: "a5439d31b6055087b3641f3039ad8f00"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.478072 10872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:32.506136 10872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:32.508652 10872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:32.508890 10872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:32.512749 10872 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:35057
I20260928 21:59:32.512797 10939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:35057 every 8 connection(s)
I20260928 21:59:32.513192 10872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:32.514076 10940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:32.515146 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 10872
I20260928 21:59:32.515293 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:32.515471 2709 external_mini_cluster.cc:1428] Setting key da73bd1ca9fffa361e4908a431674cf8
I20260928 21:59:32.517102 10940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Bootstrap starting.
I20260928 21:59:32.517799 10940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:32.518117 10940 log.cc:824] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:32.518905 10940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: No bootstrap required, opened a new log
I20260928 21:59:32.521026 10940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:32.521255 10940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:32.521304 10940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7886fa055fe04da5817a68490f47c300, State: Initialized, Role: FOLLOWER
I20260928 21:59:32.521466 10940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:32.521549 10940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:32.521574 10940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:32.521647 10940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:32.522289 10940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:32.522419 10940 leader_election.cc:302] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7886fa055fe04da5817a68490f47c300; no voters:
I20260928 21:59:32.522614 10940 leader_election.cc:288] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:32.522677 10945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:32.522876 10945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 LEADER]: Becoming Leader. State: Replica: 7886fa055fe04da5817a68490f47c300, State: Running, Role: LEADER
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:32.523036 10945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:32.522908 10940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:32.523434 10947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7886fa055fe04da5817a68490f47c300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } } }
I20260928 21:59:32.523517 10947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:32.523695 10946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7886fa055fe04da5817a68490f47c300. Latest consensus state: current_term: 1 leader_uuid: "7886fa055fe04da5817a68490f47c300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } } }
I20260928 21:59:32.523782 10946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:32.525092 10949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:32.525642 10949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:32.527486 10949 catalog_manager.cc:1347] Generated new cluster ID: 78faa51b5d6e4cf6b9fec3dd4b0a285d
I20260928 21:59:32.527531 10949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:32.527808 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:32.516165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.2.165.126:35057)
0928 21:59:32.516399 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:32.516409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:32.516446 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:59:32.517062 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:32.517071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:32.517139 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:32.517297 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:32.518067 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.519135 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.520087 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.520552 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.520721 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:32.523724 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:32.523729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:32.523730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:32.523744 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:32.525998 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:32.526422 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:32.526426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:32.526426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:32.526540 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:32.526783 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:32.526785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:32.526786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:32.527159 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:32.527336 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:32.527437 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 21:59:32.527593 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":114,"threads_started":1}
I20260928 21:59:32.540442 10949 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:32.541116 10949 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:32.543078 10949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Generated new TSK 0
I20260928 21:59:32.543444 10949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:59:32.572542 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35057
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:32.658878 10966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:32.659096 10966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:32.659121 10966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:32.660295 10966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:32.660337 10966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:32.660363 10966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:32.661837 10966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:32.662053 10966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:32.662250 10966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:32.667321 10972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:32.667328 10974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:32.667395 10971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:32.667572 10966 server_base.cc:1034] running on GCE node
I20260928 21:59:32.667840 10966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:32.668126 10966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:32.669315 10966 hybrid_clock.cc:648] HybridClock initialized: now 1790632772669301 us; error 43 us; skew 500 ppm
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:32.671826 10966 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:59:32.672338 10966 webserver.cc:460] Webserver started at http://127.2.165.65:41383/ using document root <none> and password file <none>
I20260928 21:59:32.672583 10966 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:32.672657 10966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:32.672828 10966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:32.673889 10966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "e5ceb63f5ed04dd59197ef642bb5bc4f"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "de60b1d7dc3ee461591d07d6852bba3d"
server_key_iv: "3d7bc8281b0bac738874eb583c4a28f1"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.674203 10966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ceb63f5ed04dd59197ef642bb5bc4f"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "de60b1d7dc3ee461591d07d6852bba3d"
server_key_iv: "3d7bc8281b0bac738874eb583c4a28f1"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.675477 10966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:32.676240 10981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.676507 10966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:32.676578 10966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "e5ceb63f5ed04dd59197ef642bb5bc4f"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "de60b1d7dc3ee461591d07d6852bba3d"
server_key_iv: "3d7bc8281b0bac738874eb583c4a28f1"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.676635 10966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:32.690567 10966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:32.693120 10966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:32.693249 10966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:32.693486 10966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:32.693840 10966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:32.693877 10966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.693900 10966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:32.693912 10966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.699749 10966 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:41541
I20260928 21:59:32.699801 11094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:41541 every 8 connection(s)
I20260928 21:59:32.700119 10966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:32.708523 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 10966
I20260928 21:59:32.708642 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:32.708849 2709 external_mini_cluster.cc:1428] Setting key f44a9bfdf614ce4b73372dfcaf019017
I20260928 21:59:32.709650 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:32.700941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:38529 (local address 127.2.165.126:35057)
0928 21:59:32.700990 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:32.700993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:32.701227 (+ 234us) server_negotiation.cc:407] Connection header received
0928 21:59:32.701693 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:32.701694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:32.701705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:32.701769 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:32.702678 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.703165 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.704109 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.704324 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.704375 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:32.706608 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:32.706609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:32.706610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:32.706618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:32.708051 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:32.708524 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:32.708525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:32.708525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:32.708588 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:32.708894 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:32.708895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:32.708896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:32.709036 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:32.709109 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:32.709424 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 21:59:32.709542 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:59:32.709859 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:32.700930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:38529)
0928 21:59:32.701104 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:32.701142 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:32.701609 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:32.701830 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:32.701834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:32.702404 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:32.702600 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:32.702604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.703248 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.703248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:32.704037 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:32.704039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.704399 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.704400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:32.704502 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:32.705027 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:32.705035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:32.706495 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:32.708183 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:32.708191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:32.708193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:32.708445 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:32.708664 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:32.708665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:32.708665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:32.708817 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:32.709113 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:32.709119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:32.709284 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 21:59:32.709397 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 21:59:32.709569 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260928 21:59:32.710664 11095 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35057
I20260928 21:59:32.710772 11095 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:32.710973 11095 heartbeater.cc:511] Master 127.2.165.126:35057 requested a full tablet report, sending...
I20260928 21:59:32.711608 10904 ts_manager.cc:194] Registered new tserver with Master: e5ceb63f5ed04dd59197ef642bb5bc4f (127.2.165.65:41541)
I20260928 21:59:32.712548 10904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:38529
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:59:32.736032 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35057
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:32.817662 11100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:32.817884 11100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:32.817909 11100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:32.819128 11100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:32.819175 11100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:32.819238 11100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:32.820756 11100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:32.821014 11100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:32.821257 11100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:32.826581 11106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:32.826581 11105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:32.826622 11108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:32.826750 11100 server_base.cc:1034] running on GCE node
I20260928 21:59:32.826995 11100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:32.827198 11100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:32.828334 11100 hybrid_clock.cc:648] HybridClock initialized: now 1790632772828328 us; error 34 us; skew 500 ppm
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:32.830888 11100 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:59:32.831380 11100 webserver.cc:460] Webserver started at http://127.2.165.66:45621/ using document root <none> and password file <none>
I20260928 21:59:32.831624 11100 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:32.831689 11100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:32.831844 11100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:32.832965 11100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "161476d507814c63acfec9dcd2cc5cb8"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "105a99ec8624e58d239987224a6e93f4"
server_key_iv: "adfceb1d7e4daa3581e26b32e230dab6"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.833359 11100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "161476d507814c63acfec9dcd2cc5cb8"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "105a99ec8624e58d239987224a6e93f4"
server_key_iv: "adfceb1d7e4daa3581e26b32e230dab6"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.834657 11100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:32.835381 11115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.835675 11100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:32.835781 11100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "161476d507814c63acfec9dcd2cc5cb8"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "105a99ec8624e58d239987224a6e93f4"
server_key_iv: "adfceb1d7e4daa3581e26b32e230dab6"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.835855 11100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:32.847409 11100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:32.849464 11100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:32.849640 11100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:32.849880 11100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:32.850207 11100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:32.850263 11100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.850337 11100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:32.850359 11100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.856199 11100 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:36075
I20260928 21:59:32.856236 11228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:36075 every 8 connection(s)
I20260928 21:59:32.856560 11100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:32.861253 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 11100
I20260928 21:59:32.861382 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:32.861565 2709 external_mini_cluster.cc:1428] Setting key 3a70b3c6ac0ecfa709b3ad086044b9de
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:32.867255 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:32.857403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:38659 (local address 127.2.165.126:35057)
0928 21:59:32.857459 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:32.857461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:32.857647 (+ 186us) server_negotiation.cc:407] Connection header received
0928 21:59:32.858235 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:32.858237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:32.858250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:32.858321 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:32.859326 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.859824 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.860874 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.861094 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.861182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:32.863653 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:32.863656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:32.863656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:32.863667 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:32.865572 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:32.866035 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:32.866036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:32.866037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:32.866104 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:32.866451 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:32.866452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:32.866452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:32.866619 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:32.866699 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:32.866994 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 21:59:32.867097 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:59:32.867302 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:32.857344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:38659)
0928 21:59:32.857503 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:32.857554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:32.858119 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:32.858403 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:32.858407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:32.859006 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:32.859215 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:32.859219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.859926 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.859927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:32.860780 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:32.860782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:32.861189 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:32.861190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:32.861337 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:32.861985 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:32.861996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:32.863489 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:32.865669 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:32.865676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:32.865679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:32.865946 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:32.866211 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:32.866212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:32.866212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:32.866375 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:32.866698 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:32.866704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:32.866850 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 21:59:32.866948 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 21:59:32.867087 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260928 21:59:32.867904 11229 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35057
I20260928 21:59:32.868010 11229 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:32.868225 11229 heartbeater.cc:511] Master 127.2.165.126:35057 requested a full tablet report, sending...
I20260928 21:59:32.868718 10904 ts_manager.cc:194] Registered new tserver with Master: 161476d507814c63acfec9dcd2cc5cb8 (127.2.165.66:36075)
I20260928 21:59:32.869089 10904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:38659
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:59:32.893132 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35057
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:32.974517 11234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:32.974802 11234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:32.974846 11234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:32.976058 11234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:32.976138 11234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:32.976181 11234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:32.977801 11234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:32.978076 11234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:32.978309 11234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:32.983646 11239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:32.983845 11240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:32.983929 11242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:32.983915 11234 server_base.cc:1034] running on GCE node
I20260928 21:59:32.984246 11234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:32.984494 11234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:32.985631 11234 hybrid_clock.cc:648] HybridClock initialized: now 1790632772985639 us; error 50 us; skew 500 ppm
Sep 28 21:59:32 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:32.988246 11234 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:59:32.988766 11234 webserver.cc:460] Webserver started at http://127.2.165.67:46047/ using document root <none> and password file <none>
I20260928 21:59:32.988997 11234 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:32.989053 11234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:32.989148 11234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:32.990159 11234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "f8e94e0330cf4ea8af1a745605dd97ec"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "1d98db68775eb1c6517547c301bc9f53"
server_key_iv: "fa8e871810b9232fdd935805d433b956"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.990473 11234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "f8e94e0330cf4ea8af1a745605dd97ec"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "1d98db68775eb1c6517547c301bc9f53"
server_key_iv: "fa8e871810b9232fdd935805d433b956"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.991835 11234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:32.992714 11249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:32.992913 11234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:32.993016 11234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "f8e94e0330cf4ea8af1a745605dd97ec"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "1d98db68775eb1c6517547c301bc9f53"
server_key_iv: "fa8e871810b9232fdd935805d433b956"
server_key_version: "encryptionkey@0"
I20260928 21:59:32.993108 11234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:33.013176 11234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:33.015503 11234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:33.015677 11234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:33.015980 11234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:33.016327 11234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:33.016382 11234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.016418 11234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:33.016465 11234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.022408 11234 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:44651
I20260928 21:59:33.022449 11362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:44651 every 8 connection(s)
I20260928 21:59:33.022814 11234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:33.028417 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 11234
I20260928 21:59:33.028517 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:33.028751 2709 external_mini_cluster.cc:1428] Setting key 37b2f1425d749bec7b5f6de92b96b579
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632772, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:33.033253 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.023606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:54365 (local address 127.2.165.126:35057)
0928 21:59:33.023677 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.023679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.024031 (+ 352us) server_negotiation.cc:407] Connection header received
0928 21:59:33.024566 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.024568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.024578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.024820 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.025811 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.026317 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.027412 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.027611 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.027678 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.030420 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:33.030421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:33.030422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:33.030428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:33.031965 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:33.032335 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:33.032336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:33.032336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:33.032388 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:33.032611 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:33.032612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:33.032612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:33.032811 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:33.032866 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:33.033114 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 21:59:33.033188 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:59:33.033447 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.023566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:54365)
0928 21:59:33.023776 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.023883 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.024481 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.024889 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.024893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.025502 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.025718 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.025723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.026419 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.026420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.027305 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.027308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.027730 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.027732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.027918 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.028625 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.028634 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.030333 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:33.032038 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:33.032044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:33.032046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:33.032266 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:33.032449 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:33.032450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:33.032450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:33.032556 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:33.032875 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:33.032880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:33.033000 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 21:59:33.033088 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:59:33.033210 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260928 21:59:33.033977 11363 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35057
I20260928 21:59:33.034112 11363 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:33.034310 11363 heartbeater.cc:511] Master 127.2.165.126:35057 requested a full tablet report, sending...
I20260928 21:59:33.034730 10904 ts_manager.cc:194] Registered new tserver with Master: f8e94e0330cf4ea8af1a745605dd97ec (127.2.165.67:44651)
I20260928 21:59:33.035109 10904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:54365
I20260928 21:59:33.040644 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab.
Entry for principal oryx/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab.
WARNING: no policy specified for oryx/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab.
Entry for principal oryx/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab.
WARNING: no policy specified for oryx/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab.
Entry for principal oryx/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab.
I20260928 21:59:33.115466 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 10966
I20260928 21:59:33.120225 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 11100
I20260928 21:59:33.125068 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 11234
I20260928 21:59:33.129240 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 10872
I20260928 21:59:33.134721 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35057
--webserver_interface=127.2.165.126
--webserver_port=45797
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:35057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:33.216815 11372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:33.217037 11372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:33.217061 11372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:33.218267 11372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:33.218314 11372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:33.218328 11372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:33.218343 11372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:33.218355 11372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:33.219909 11372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:35057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=45797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:33.220218 11372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:33.220482 11372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:33.225497 11378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:33.225587 11380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:33.225564 11377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:33.225811 11372 server_base.cc:1034] running on GCE node
I20260928 21:59:33.226090 11372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:33.226392 11372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:33.227586 11372 hybrid_clock.cc:648] HybridClock initialized: now 1790632773227565 us; error 39 us; skew 500 ppm
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632773, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:33.230455 11372 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:33.230919 11372 webserver.cc:460] Webserver started at http://127.2.165.126:45797/ using document root <none> and password file <none>
I20260928 21:59:33.231176 11372 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:33.231295 11372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:33.232811 11372 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:59:33.233546 11387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.233736 11372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:33.233880 11372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "7886fa055fe04da5817a68490f47c300"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "f059973683d5d01c3463228e1b4d66d2"
server_key_iv: "a5439d31b6055087b3641f3039ad8f00"
server_key_version: "encryptionkey@0"
I20260928 21:59:33.234251 11372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:33.246920 11372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:33.248952 11372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:33.249141 11372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:33.253048 11372 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:35057
I20260928 21:59:33.253083 11439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:35057 every 8 connection(s)
I20260928 21:59:33.253480 11372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:33.254979 11440 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:59:33.255643 11440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Bootstrap starting.
I20260928 21:59:33.259760 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 11372
I20260928 21:59:33.260047 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:41541
--local_ip_for_outbound_sockets=127.2.165.65
--tserver_master_addrs=127.2.165.126:35057
--webserver_port=41383
--webserver_interface=127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:59:33.264868 11440 log.cc:824] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:33.266621 11440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:59:33.266923 11440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Bootstrap complete.
I20260928 21:59:33.269322 11440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:33.269577 11440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7886fa055fe04da5817a68490f47c300, State: Initialized, Role: FOLLOWER
I20260928 21:59:33.269796 11440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:33.269893 11440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:33.269948 11440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:33.269997 11440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:59:33.271008 11440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:33.271126 11440 leader_election.cc:302] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7886fa055fe04da5817a68490f47c300; no voters:
I20260928 21:59:33.271332 11440 leader_election.cc:288] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:59:33.271368 11444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:59:33.271481 11444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [term 2 LEADER]: Becoming Leader. State: Replica: 7886fa055fe04da5817a68490f47c300, State: Running, Role: LEADER
I20260928 21:59:33.271613 11440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:33.271636 11444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } }
I20260928 21:59:33.272015 11445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7886fa055fe04da5817a68490f47c300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } } }
I20260928 21:59:33.272099 11445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:33.272411 11446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7886fa055fe04da5817a68490f47c300. Latest consensus state: current_term: 2 leader_uuid: "7886fa055fe04da5817a68490f47c300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7886fa055fe04da5817a68490f47c300" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35057 } } }
I20260928 21:59:33.272454 11450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:33.272549 11446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:33.273120 11450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:33.273411 11450 catalog_manager.cc:1259] Loaded cluster ID: 78faa51b5d6e4cf6b9fec3dd4b0a285d
I20260928 21:59:33.273454 11450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:33.274648 11450 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:33.274828 11450 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7886fa055fe04da5817a68490f47c300: Loaded TSK: 0
I20260928 21:59:33.275033 11450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:59:33.344156 11442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:33.344369 11442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:33.344394 11442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:33.345695 11442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:33.345741 11442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:33.345769 11442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:33.347296 11442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.65.keytab
--principal=oryx/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:41541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=41383
--tserver_master_addrs=127.2.165.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:33.347553 11442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:33.347810 11442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:33.353734 11469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:33.353729 11466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:33.353878 11467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:33.354223 11442 server_base.cc:1034] running on GCE node
I20260928 21:59:33.354463 11442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:33.354748 11442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:33.355913 11442 hybrid_clock.cc:648] HybridClock initialized: now 1790632773355896 us; error 36 us; skew 500 ppm
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632773, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:33.358803 11442 init.cc:377] Logged in from keytab as oryx/127.2.165.65@KRBTEST.COM (short username oryx)
I20260928 21:59:33.359347 11442 webserver.cc:460] Webserver started at http://127.2.165.65:41383/ using document root <none> and password file <none>
I20260928 21:59:33.359625 11442 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:33.359706 11442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:33.361658 11442 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:59:33.362355 11476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.362576 11442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:33.362673 11442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "e5ceb63f5ed04dd59197ef642bb5bc4f"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "de60b1d7dc3ee461591d07d6852bba3d"
server_key_iv: "3d7bc8281b0bac738874eb583c4a28f1"
server_key_version: "encryptionkey@0"
I20260928 21:59:33.363215 11442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:33.382673 11442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:33.385251 11442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:33.385414 11442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:33.385674 11442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:33.386037 11442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:33.386086 11442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.386124 11442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:33.386142 11442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.393567 11442 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:41541
I20260928 21:59:33.393599 11589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:41541 every 8 connection(s)
I20260928 21:59:33.394061 11442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:33.395918 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 11442
I20260928 21:59:33.396261 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:36075
--local_ip_for_outbound_sockets=127.2.165.66
--tserver_master_addrs=127.2.165.126:35057
--webserver_port=45621
--webserver_interface=127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.404525 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.394778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:52011)
0928 21:59:33.394980 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.395026 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.395611 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.396197 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.396202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.396952 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.397216 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.397224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.398778 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.398779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.400017 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.400020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.400610 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.400611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.400790 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.401337 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.401348 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.404266 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
W20260928 21:59:33.404874 11590 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:35057 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.165.126:35057: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
W20260928 21:59:33.405126 11593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.394891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:52011 (local address 127.2.165.126:35057)
0928 21:59:33.395076 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.395092 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.395121 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:59:33.395918 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.395927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.395992 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.396152 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.397351 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.398647 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.400103 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.400517 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.400599 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.404502 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:52011: BlockingRecv error: recv got EOF from 127.2.165.65:52011 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.411010 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.405103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:45689)
0928 21:59:33.405203 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.405214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.405258 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.405651 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.405652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.405830 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.405896 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.405898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.406869 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.406871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.407801 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.407804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.408290 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.408291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.408373 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.408909 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.408915 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.410938 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 21:59:33.411300 11593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.405219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:45689 (local address 127.2.165.126:35057)
0928 21:59:33.405493 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.405495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.405505 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:33.405535 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.405536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.405550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.405624 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.405998 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.406774 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.407897 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.408201 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.408273 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.411239 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:45689: BlockingRecv error: recv got EOF from 127.2.165.65:45689 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":83,"threads_started":1}
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.416406 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.411251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:41091)
0928 21:59:33.411371 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.411381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.411417 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.411635 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.411635 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.411784 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.411819 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.411820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.412759 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.412761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.413410 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.413413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.413914 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.413916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.413973 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.414448 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.414453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.416307 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260928 21:59:33.416437 11595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.411338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:41091 (local address 127.2.165.126:35057)
0928 21:59:33.411504 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.411506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.411512 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:59:33.411534 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.411534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.411544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.411599 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.411891 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.412648 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.413498 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.413834 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.413899 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.416393 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:41091: BlockingRecv error: recv got EOF from 127.2.165.65:41091 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260928 21:59:33.416581 11590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:59:33.485249 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:33.485471 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:33.485495 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:33.486701 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:33.486744 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:33.486771 11594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:33.488163 11594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.66.keytab
--principal=oryx/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:36075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=45621
--tserver_master_addrs=127.2.165.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:33.488385 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:33.488565 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:33.494081 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:33.494123 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:33.494081 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:33.494172 11594 server_base.cc:1034] running on GCE node
I20260928 21:59:33.494550 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:33.494772 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:33.495936 11594 hybrid_clock.cc:648] HybridClock initialized: now 1790632773495917 us; error 40 us; skew 500 ppm
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632773, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:33.498808 11594 init.cc:377] Logged in from keytab as oryx/127.2.165.66@KRBTEST.COM (short username oryx)
I20260928 21:59:33.499269 11594 webserver.cc:460] Webserver started at http://127.2.165.66:45621/ using document root <none> and password file <none>
I20260928 21:59:33.499501 11594 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:33.499558 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:33.501085 11594 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:59:33.501689 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.501875 11594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:59:33.501946 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "161476d507814c63acfec9dcd2cc5cb8"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "105a99ec8624e58d239987224a6e93f4"
server_key_iv: "adfceb1d7e4daa3581e26b32e230dab6"
server_key_version: "encryptionkey@0"
I20260928 21:59:33.502314 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:33.516309 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:33.518677 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:33.518837 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:33.519094 11594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:33.519410 11594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:33.519443 11594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.519465 11594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:33.519512 11594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.525705 11594 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:36075
I20260928 21:59:33.525764 11723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:36075 every 8 connection(s)
I20260928 21:59:33.526108 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.533844 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.526924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:35313)
0928 21:59:33.527086 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.527129 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.527607 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.527828 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.527831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.528410 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.528607 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.528612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.529450 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.529451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.530237 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.530239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.530590 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.530591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.530690 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.531276 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.531285 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.533684 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
W20260928 21:59:33.533867 11595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.526912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:35313 (local address 127.2.165.126:35057)
0928 21:59:33.526992 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.526994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.527229 (+ 235us) server_negotiation.cc:407] Connection header received
0928 21:59:33.527696 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.527697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.527709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.527773 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.528804 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.529364 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.530327 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.530516 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.530574 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.533792 (+ 3218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:35313: BlockingRecv error: recv got EOF from 127.2.165.66:35313 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 21:59:33.534251 11724 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:35057 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.165.126:35057: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
I20260928 21:59:33.534402 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 11594
I20260928 21:59:33.534741 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:44651
--local_ip_for_outbound_sockets=127.2.165.67
--tserver_master_addrs=127.2.165.126:35057
--webserver_port=46047
--webserver_interface=127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.539994 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.534593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:57391)
0928 21:59:33.534724 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.534736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.534784 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.534953 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.534954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.535109 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.535179 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.535181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.535862 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.535864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.536496 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.536498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.537055 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.537056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.537155 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.537619 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.537625 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.539875 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260928 21:59:33.540023 11595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.534701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:57391 (local address 127.2.165.126:35057)
0928 21:59:33.534756 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.534758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.534772 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:59:33.534843 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.534844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.534857 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.534933 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.535280 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.535776 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.536595 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.536975 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.537053 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.539976 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:57391: BlockingRecv error: recv got EOF from 127.2.165.66:57391 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.546010 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.540425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:56329)
0928 21:59:33.540477 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.540489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.540544 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.540711 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.540712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.540881 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.540926 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.540928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.541564 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.541565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.542525 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.542529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.543091 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.543093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.543158 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.543724 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.543731 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.545901 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 21:59:33.546047 11595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.540500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:56329 (local address 127.2.165.126:35057)
0928 21:59:33.540548 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.540549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.540558 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:33.540603 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.540603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.540613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.540659 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.541021 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.541482 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.542632 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.543003 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.543099 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.546001 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:56329: BlockingRecv error: recv got EOF from 127.2.165.66:56329 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 21:59:33.546250 11724 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:59:33.620621 11727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:33.620893 11727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:33.620934 11727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:33.622144 11727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:33.622215 11727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:33.622267 11727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:33.623752 11727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/krb5kdc/oryx_127.2.165.67.keytab
--principal=oryx/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:44651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=46047
--tserver_master_addrs=127.2.165.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:33.624023 11727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:33.624257 11727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:33.629462 11732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:33.629462 11735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:33.630244 11727 server_base.cc:1034] running on GCE node
W20260928 21:59:33.629545 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:33.630723 11727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:33.631055 11727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:33.632210 11727 hybrid_clock.cc:648] HybridClock initialized: now 1790632773632197 us; error 28 us; skew 500 ppm
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632773, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:33.634776 11727 init.cc:377] Logged in from keytab as oryx/127.2.165.67@KRBTEST.COM (short username oryx)
I20260928 21:59:33.635239 11727 webserver.cc:460] Webserver started at http://127.2.165.67:46047/ using document root <none> and password file <none>
I20260928 21:59:33.635480 11727 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:33.635561 11727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:33.637046 11727 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:33.637773 11742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.637952 11727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:33.638059 11727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "f8e94e0330cf4ea8af1a745605dd97ec"
format_stamp: "Formatted at 2026-09-28 21:59:32 on dist-test-slave-3t36"
server_key: "1d98db68775eb1c6517547c301bc9f53"
server_key_iv: "fa8e871810b9232fdd935805d433b956"
server_key_version: "encryptionkey@0"
I20260928 21:59:33.638437 11727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:33.658582 11727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:33.660784 11727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:33.660950 11727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:33.661197 11727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:33.661522 11727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:33.661576 11727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.661619 11727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:33.661649 11727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:33.667994 11727 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:44651
I20260928 21:59:33.668131 11855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:44651 every 8 connection(s)
I20260928 21:59:33.668473 11727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:33.670729 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 11727
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:59:33.677440 11595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.669332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:32781 (local address 127.2.165.126:35057)
0928 21:59:33.669376 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.669378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.669591 (+ 213us) server_negotiation.cc:407] Connection header received
0928 21:59:33.670078 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.670080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.670091 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.670155 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.671164 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.672055 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.673383 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.673678 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.673751 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.677384 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:32781: BlockingRecv error: recv got EOF from 127.2.165.67:32781 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:59:33.677485 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.669297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:32781)
0928 21:59:33.669454 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.669499 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.669980 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.670211 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.670214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.670836 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.671062 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.671068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.672158 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.672160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.673302 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.673305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.673824 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.673825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.673987 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.674636 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.674648 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.677239 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
W20260928 21:59:33.677703 11856 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:35057 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.165.126:35057: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
I20260928 21:59:33.679448 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.671289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.2.165.126:35057)
0928 21:59:33.671369 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.671371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.671381 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:33.671407 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.671408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.671420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.671483 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.672216 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.672770 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.673704 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.674005 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.674087 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.675590 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:33.675595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:33.675595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:33.675607 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:33.677683 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:33.678247 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:33.678251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:33.678251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:33.678389 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:33.678604 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:33.678605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:33.678605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:33.678918 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:33.679061 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:33.679224 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:59:33.679340 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.683210 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.677986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:60569)
0928 21:59:33.678036 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.678048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.678102 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.678289 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.678290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.678473 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.678540 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.678542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.679474 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.679476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.680252 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.680255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.680608 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.680609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.680659 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.681090 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.681094 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.683129 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:59:33.683297 11595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.678040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:60569 (local address 127.2.165.126:35057)
0928 21:59:33.678097 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.678098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.678106 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:59:33.678165 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.678166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.678173 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.678242 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.678648 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.679388 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.680339 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.680546 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.680598 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.683236 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:60569: BlockingRecv error: recv got EOF from 127.2.165.67:60569 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:33 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:33.688288 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:33.683728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:40725)
0928 21:59:33.683797 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:33.683808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:33.683855 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:33.684007 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:33.684008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:33.684113 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:33.684140 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.684141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.685051 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.685052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.685620 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:33.685622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.685962 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.685963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:33.685998 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.686365 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:33.686369 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:33.688230 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 21:59:33.688524 11595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:33.683827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:40725 (local address 127.2.165.126:35057)
0928 21:59:33.683896 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:33.683898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:33.683906 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:59:33.683933 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:33.683934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:33.683942 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:33.683991 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:33.684370 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.684979 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.685713 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:33.685890 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:33.685956 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:33.688343 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:40725: BlockingRecv error: recv got EOF from 127.2.165.67:40725 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 21:59:33.688585 11856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 21:59:34 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:34 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:34.421977 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:34.417016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:34963)
0928 21:59:34.417193 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:34.417204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:34.417270 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:34.417445 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:34.417446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:34.417608 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:34.417657 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:34.417659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.418361 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.418362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:34.418938 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:34.418940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.419331 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.419332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:34.419390 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:34.419920 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:34.419925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:34.421904 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
W20260928 21:59:34.422151 11861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:34.417109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:34963 (local address 127.2.165.126:35057)
0928 21:59:34.417221 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:34.417223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:34.417253 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:59:34.417351 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:34.417351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:34.417361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:34.417420 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:34.417723 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.418265 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.419029 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.419222 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.419275 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:34.422050 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:34963: BlockingRecv error: recv got EOF from 127.2.165.65:34963 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
Sep 28 21:59:34 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:34 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:34.551836 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:34.546774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:51273)
0928 21:59:34.546913 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:34.546926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:34.547005 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:34.547185 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:34.547186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:34.547369 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:34.547453 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:34.547455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.548210 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.548211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:34.548791 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:34.548793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.549171 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.549171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:34.549216 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:34.549658 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:34.549663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:34.551759 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260928 21:59:34.551944 11861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:34.546865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:51273 (local address 127.2.165.126:35057)
0928 21:59:34.546922 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:34.546925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:34.547007 (+ 82us) server_negotiation.cc:407] Connection header received
0928 21:59:34.547076 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:34.547077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:34.547092 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:34.547155 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:34.547564 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.548131 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.548911 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.549098 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.549155 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:34.551873 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:51273: BlockingRecv error: recv got EOF from 127.2.165.66:51273 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 21:59:34 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:34 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:34.693985 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:34.689122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:54627)
0928 21:59:34.689266 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:34.689280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:34.689364 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:34.689552 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:34.689553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:34.689775 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:34.689825 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:34.689827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.690520 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.690522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:34.691079 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:34.691081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.691450 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.691451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:34.691516 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:34.692004 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:34.692010 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:34.693917 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
W20260928 21:59:34.694119 11861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:34.689247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:54627 (local address 127.2.165.126:35057)
0928 21:59:34.689304 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:34.689306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:34.689337 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:59:34.689454 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:34.689455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:34.689471 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:34.689543 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:34.689937 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.690434 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.691171 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:34.691368 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:34.691440 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:34.694015 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:54627: BlockingRecv error: recv got EOF from 127.2.165.67:54627 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 21:59:35 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:35 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:35.427274 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:35.422462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:55577)
0928 21:59:35.422607 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:35.422618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:35.422669 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:35.422881 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:35.422882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:35.423053 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:35.423103 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:35.423105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.423815 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.423818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:35.424379 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:35.424380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.424843 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.424844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:35.424895 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:35.425369 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:35.425375 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:35.427192 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260928 21:59:35.427434 11865 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:35.422567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:55577 (local address 127.2.165.126:35057)
0928 21:59:35.422722 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:35.422723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:35.422737 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:59:35.422762 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:35.422762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:35.422773 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:35.422840 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:35.423213 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.423721 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.424469 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.424738 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.424821 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:35.427347 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:55577: BlockingRecv error: recv got EOF from 127.2.165.65:55577 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
Sep 28 21:59:35 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:35 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:35.557293 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:35.552335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:55859)
0928 21:59:35.552509 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:35.552521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:35.552593 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:35.552749 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:35.552750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:35.552944 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:35.552990 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:35.552992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.553665 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.553667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:35.554211 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:35.554213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.554596 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.554597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:35.554654 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:35.555171 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:35.555177 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:35.557229 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
W20260928 21:59:35.557524 11865 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:35.552425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:55859 (local address 127.2.165.126:35057)
0928 21:59:35.552504 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:35.552506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:35.552572 (+ 66us) server_negotiation.cc:407] Connection header received
0928 21:59:35.552655 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:35.552656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:35.552696 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:35.552746 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:35.553077 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.553580 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.554302 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.554503 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.554593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:35.557461 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:55859: BlockingRecv error: recv got EOF from 127.2.165.66:55859 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 21:59:35 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:35 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:35.699645 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:35.694692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:39371)
0928 21:59:35.694815 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:35.694834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:35.694922 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:35.695117 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:35.695118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:35.695333 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:35.695409 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:35.695411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.696117 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.696118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:35.696709 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:35.696711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.697070 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.697071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:35.697115 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:35.697643 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:35.697648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:35.699585 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
W20260928 21:59:35.699786 11865 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:35.694903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:39371 (local address 127.2.165.126:35057)
0928 21:59:35.694956 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:35.694958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:35.694973 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:59:35.695011 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:35.695011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:35.695025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:35.695090 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:35.695522 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.696038 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.696800 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:35.696994 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:35.697055 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:35.699696 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:39371: BlockingRecv error: recv got EOF from 127.2.165.67:39371 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 21:59:36 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:36 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:36.432979 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:36.427811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:51659)
0928 21:59:36.427949 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:36.427960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:36.428018 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:36.428302 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:36.428304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:36.428538 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:36.428602 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:36.428604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.429375 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.429376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:36.429951 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:36.429953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.430284 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.430284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:36.430323 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:36.430820 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:36.430826 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:36.432878 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260928 21:59:36.433040 11869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:36.427920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:51659 (local address 127.2.165.126:35057)
0928 21:59:36.428068 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:36.428070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:36.428079 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:36.428115 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:36.428116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:36.428146 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:36.428228 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:36.428715 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.429297 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.430030 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.430219 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.430276 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:36.432964 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:51659: BlockingRecv error: recv got EOF from 127.2.165.65:51659 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
Sep 28 21:59:36 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:36 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:36.563138 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:36.558035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:41083)
0928 21:59:36.558218 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:36.558229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:36.558296 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:36.558488 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:36.558489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:36.558672 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:36.558748 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:36.558750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.559505 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.559506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:36.560045 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:36.560047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.560431 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.560432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:36.560475 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:36.561073 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:36.561079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:36.563079 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
W20260928 21:59:36.563292 11869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:36.558111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:41083 (local address 127.2.165.126:35057)
0928 21:59:36.558217 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:36.558219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:36.558348 (+ 129us) server_negotiation.cc:407] Connection header received
0928 21:59:36.558386 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:36.558387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:36.558399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:36.558446 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:36.558883 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.559408 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.560140 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.560346 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.560406 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:36.563204 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:41083: BlockingRecv error: recv got EOF from 127.2.165.66:41083 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 21:59:36 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:36 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:36.705173 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:36.700175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:59313)
0928 21:59:36.700349 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:36.700360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:36.700440 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:36.700567 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:36.700567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:36.700753 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:36.700801 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:36.700803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.701502 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.701503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:36.702131 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:36.702135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.702525 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.702526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:36.702578 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:36.703027 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:36.703032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:36.705074 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
W20260928 21:59:36.705224 11869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:36.700306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:59313 (local address 127.2.165.126:35057)
0928 21:59:36.700370 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:36.700371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:36.700429 (+ 58us) server_negotiation.cc:407] Connection header received
0928 21:59:36.700495 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:36.700496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:36.700508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:36.700548 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:36.700897 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.701405 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.702234 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:36.702444 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:36.702510 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:36.705160 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:59313: BlockingRecv error: recv got EOF from 127.2.165.67:59313 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 21:59:37 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:37 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:37.439023 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:37.433541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:41055)
0928 21:59:37.433696 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:37.433707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:37.433775 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:37.433921 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:37.433922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:37.434095 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:37.434145 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:37.434147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.434874 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.434875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:37.435422 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:37.435424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.435906 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.435907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:37.435959 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:37.436565 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:37.436570 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:37.438961 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
W20260928 21:59:37.439185 11873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:37.433670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:41055 (local address 127.2.165.126:35057)
0928 21:59:37.433805 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:37.433807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:37.433817 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:37.433843 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:37.433844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:37.433854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:37.433917 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:37.434221 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.434738 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.435516 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.435741 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.435867 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:37.439096 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:41055: BlockingRecv error: recv got EOF from 127.2.165.65:41055 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 28 21:59:37 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:37 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:37.568557 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:37.563628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:51735)
0928 21:59:37.563757 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:37.563770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:37.563838 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:37.564027 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:37.564028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:37.564187 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:37.564236 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:37.564237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.564941 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.564942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:37.565486 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:37.565487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.565843 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.565844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:37.565882 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:37.566350 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:37.566355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:37.568468 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20260928 21:59:37.568804 11873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:37.563713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:51735 (local address 127.2.165.126:35057)
0928 21:59:37.563878 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:37.563879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:37.563889 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:37.563912 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:37.563913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:37.563923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:37.563981 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:37.564340 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.564875 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.565575 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.565771 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.565838 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:37.568727 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:51735: BlockingRecv error: recv got EOF from 127.2.165.66:51735 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 28 21:59:37 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:37 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:37.710664 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:37.705768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:37463)
0928 21:59:37.705924 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:37.705937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:37.706014 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:37.706153 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:37.706154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:37.706413 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:37.706474 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:37.706477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.707167 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.707168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:37.707726 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:37.707728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.708098 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.708099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:37.708154 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:37.708707 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:37.708713 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:37.710603 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
W20260928 21:59:37.710896 11873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:37.705823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:37463 (local address 127.2.165.126:35057)
0928 21:59:37.705934 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:37.705937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:37.705992 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:59:37.706077 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:37.706078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:37.706091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:37.706148 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:37.706562 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.707066 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.707816 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:37.708020 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:37.708102 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:37.710759 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:37463: BlockingRecv error: recv got EOF from 127.2.165.67:37463 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 21:59:38 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:38 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:38.451978 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:38.439671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:33031)
0928 21:59:38.439969 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:38.439982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:38.440055 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:38.440251 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:38.440253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:38.440514 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:38.440566 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:38.440568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.441456 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.441458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:38.441999 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:38.442001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.442394 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.442395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:38.442456 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:38.449553 (+ 7097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:38.449560 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:38.451918 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
W20260928 21:59:38.452147 11877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:38.439782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:33031 (local address 127.2.165.126:35057)
0928 21:59:38.439918 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:38.439921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:38.440096 (+ 175us) server_negotiation.cc:407] Connection header received
0928 21:59:38.440147 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:38.440149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:38.440168 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:38.440241 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:38.440706 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.441357 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.442086 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.442301 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.442393 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:38.452061 (+ 9668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:33031: BlockingRecv error: recv got EOF from 127.2.165.65:33031 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":51,"threads_started":1}
Sep 28 21:59:38 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:38 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:38.574968 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:38.569343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:41459)
0928 21:59:38.569556 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:38.569571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:38.569640 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:38.569821 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:38.569822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:38.570028 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:38.570076 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:38.570077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.570742 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.570743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:38.571300 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:38.571301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.571758 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.571759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:38.571854 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:38.572595 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:38.572600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:38.574872 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260928 21:59:38.575031 11877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:38.569537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:41459 (local address 127.2.165.126:35057)
0928 21:59:38.569691 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:38.569693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:38.569702 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:38.569730 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:38.569731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:38.569742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:38.569803 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:38.570155 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.570646 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.571387 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.571661 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.571733 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:38.574968 (+ 3235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:41459: BlockingRecv error: recv got EOF from 127.2.165.66:41459 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 28 21:59:38 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:38 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:38.716359 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:38.711230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:33755)
0928 21:59:38.711446 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:38.711461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:38.711569 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:38.711733 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:38.711734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:38.711967 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:38.712017 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:38.712019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.712780 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.712782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:38.713320 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:38.713322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.713680 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.713681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:38.713738 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:38.714241 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:38.714246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:38.716249 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":117,"threads_started":1}
W20260928 21:59:38.716650 11877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:38.711390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:33755 (local address 127.2.165.126:35057)
0928 21:59:38.711464 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:38.711466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:38.711547 (+ 81us) server_negotiation.cc:407] Connection header received
0928 21:59:38.711639 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:38.711640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:38.711661 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:38.711726 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:38.712106 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.712631 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.713409 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:38.713604 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:38.713683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:38.716560 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:33755: BlockingRecv error: recv got EOF from 127.2.165.67:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 21:59:39 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:39 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:39.457764 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:39.452517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:51883)
0928 21:59:39.452726 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:39.452740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:39.452812 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:39.452997 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:39.452998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:39.453177 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:39.453225 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:39.453227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.453934 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.453935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:39.454489 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:39.454491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.454888 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.454889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:39.454936 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:39.455476 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:39.455481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:39.457648 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
W20260928 21:59:39.457845 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:39.452562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:51883 (local address 127.2.165.126:35057)
0928 21:59:39.452705 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:39.452707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:39.452816 (+ 109us) server_negotiation.cc:407] Connection header received
0928 21:59:39.452905 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:39.452906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:39.452918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:39.452980 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:39.453313 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.453824 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.454576 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.454759 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.454811 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:39.457751 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:51883: BlockingRecv error: recv got EOF from 127.2.165.65:51883 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
Sep 28 21:59:39 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:39 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:39.580554 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:39.575523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:37831)
0928 21:59:39.575727 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:39.575740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:39.575837 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:39.576035 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:39.576036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:39.576225 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:39.576272 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:39.576274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.577066 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.577068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:39.577611 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:39.577612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.578019 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.578020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:39.578078 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:39.578583 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:39.578588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:39.580494 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
W20260928 21:59:39.580757 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:39.575624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:37831 (local address 127.2.165.126:35057)
0928 21:59:39.575720 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:39.575722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:39.575816 (+ 94us) server_negotiation.cc:407] Connection header received
0928 21:59:39.575934 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:39.575936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:39.575949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:39.576031 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:39.576361 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.576975 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.577700 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.577933 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.578014 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:39.580663 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:37831: BlockingRecv error: recv got EOF from 127.2.165.66:37831 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 21:59:39 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:39 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:39.722329 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:39.717033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:54779)
0928 21:59:39.717205 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:39.717217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:39.717288 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:39.717472 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:39.717473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:39.717695 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:39.717757 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:39.717759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.718614 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.718615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:39.719166 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:39.719168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.719608 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.719609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:39.719670 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:39.720180 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:39.720185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:39.722225 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
W20260928 21:59:39.722426 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:39.717144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:54779 (local address 127.2.165.126:35057)
0928 21:59:39.717217 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:39.717219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:39.717289 (+ 70us) server_negotiation.cc:407] Connection header received
0928 21:59:39.717336 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:39.717337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:39.717348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:39.717410 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:39.717925 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.718532 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.719269 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:39.719517 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:39.719607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:39.722320 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:54779: BlockingRecv error: recv got EOF from 127.2.165.67:54779 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 21:59:40 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:40 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:40.463378 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:40.458339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:46947)
0928 21:59:40.458527 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:40.458538 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:40.458601 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:40.458775 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:40.458776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:40.459008 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:40.459057 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:40.459058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.459721 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.459722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:40.460296 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:40.460298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.460898 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.460899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:40.460947 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:40.461408 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:40.461413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:40.463315 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
W20260928 21:59:40.463584 11885 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:40.458455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:46947 (local address 127.2.165.126:35057)
0928 21:59:40.458601 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:40.458603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:40.458612 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:40.458678 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:40.458679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:40.458689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:40.458750 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:40.459131 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.459637 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.460409 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.460623 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.460770 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:40.463500 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:46947: BlockingRecv error: recv got EOF from 127.2.165.65:46947 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
Sep 28 21:59:40 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:40 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:40.586238 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:40.581152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:59901)
0928 21:59:40.581331 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:40.581349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:40.581411 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:40.581564 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:40.581565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:40.581729 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:40.581778 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:40.581779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.582406 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.582407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:40.582958 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:40.582960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.583302 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.583302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:40.583344 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:40.583895 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:40.583900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:40.586123 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
W20260928 21:59:40.586274 11885 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:40.581238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:59901 (local address 127.2.165.126:35057)
0928 21:59:40.581322 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:40.581330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:40.581412 (+ 82us) server_negotiation.cc:407] Connection header received
0928 21:59:40.581480 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:40.581481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:40.581496 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:40.581542 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:40.581843 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.582332 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.583056 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.583236 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.583290 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:40.586206 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:59901: BlockingRecv error: recv got EOF from 127.2.165.66:59901 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 21:59:40 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:40 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:40.727711 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:40.722987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:48873)
0928 21:59:40.723123 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:40.723142 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:40.723194 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:40.723363 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:40.723364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:40.723542 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:40.723590 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:40.723592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.724263 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.724264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:40.724885 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:40.724887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.725222 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.725223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:40.725266 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:40.725773 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:40.725777 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:40.727650 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
W20260928 21:59:40.727836 11885 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:40.723179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:48873 (local address 127.2.165.126:35057)
0928 21:59:40.723229 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:40.723233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:40.723247 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:59:40.723273 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:40.723273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:40.723284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:40.723346 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:40.723661 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.724184 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.724972 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:40.725153 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:40.725210 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:40.727752 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:48873: BlockingRecv error: recv got EOF from 127.2.165.67:48873 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 21:59:41 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:41 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:41.468991 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:41.464017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:53851)
0928 21:59:41.464199 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:41.464213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:41.464286 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:41.464500 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:41.464501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:41.464702 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:41.464751 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:41.464753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.465481 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.465482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:41.466013 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:41.466058 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.466394 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.466395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:41.466437 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:41.466973 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:41.466993 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:41.468930 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
W20260928 21:59:41.469194 11889 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:41.464175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:53851 (local address 127.2.165.126:35057)
0928 21:59:41.464347 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:41.464349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:41.464366 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:59:41.464391 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:41.464392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:41.464403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:41.464480 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:41.464828 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.465409 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.466137 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.466326 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.466384 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:41.469119 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:53851: BlockingRecv error: recv got EOF from 127.2.165.65:53851 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
Sep 28 21:59:41 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:41 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:41.591856 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:41.586771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:43735)
0928 21:59:41.586964 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:41.586977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:41.587076 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:41.587216 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:41.587217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:41.587401 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:41.587448 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:41.587450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.588299 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.588300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:41.588928 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:41.588930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.589314 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.589314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:41.589378 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:41.589903 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:41.589908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:41.591753 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":102,"threads_started":1}
W20260928 21:59:41.591928 11889 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:41.586869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:43735 (local address 127.2.165.126:35057)
0928 21:59:41.586957 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:41.586959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:41.587071 (+ 112us) server_negotiation.cc:407] Connection header received
0928 21:59:41.587140 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:41.587141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:41.587154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:41.587201 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:41.587530 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.588204 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.589030 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.589234 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.589303 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:41.591844 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:43735: BlockingRecv error: recv got EOF from 127.2.165.66:43735 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 21:59:41 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:41 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:41.733110 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:41.728218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:38029)
0928 21:59:41.728419 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:41.728433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:41.728508 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:41.728665 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:41.728691 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:41.728874 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:41.728928 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:41.728930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.729626 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.729627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:41.730216 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:41.730218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.730593 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.730594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:41.730654 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:41.731167 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:41.731179 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:41.733014 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
W20260928 21:59:41.733162 11889 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:41.728318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:38029 (local address 127.2.165.126:35057)
0928 21:59:41.728410 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:41.728412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:41.728488 (+ 76us) server_negotiation.cc:407] Connection header received
0928 21:59:41.728579 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:41.728579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:41.728597 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:41.728659 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:41.729021 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.729535 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.730309 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:41.730508 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:41.730594 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:41.733097 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:38029: BlockingRecv error: recv got EOF from 127.2.165.67:38029 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 21:59:42 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:42 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:42.474565 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:42.469472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:50859)
0928 21:59:42.469626 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:42.469639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:42.469741 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:42.469906 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:42.469907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:42.470133 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:42.470186 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:42.470187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.470899 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.470900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:42.471478 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:42.471479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.471920 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.471921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:42.471979 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:42.472578 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:42.472583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:42.474497 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
W20260928 21:59:42.474735 11893 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:42.469588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:50859 (local address 127.2.165.126:35057)
0928 21:59:42.469750 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:42.469752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:42.469783 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:59:42.469808 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:42.469809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:42.469819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:42.469899 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:42.470274 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.470804 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.471621 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.471830 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.471919 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:42.474624 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:50859: BlockingRecv error: recv got EOF from 127.2.165.65:50859 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
Sep 28 21:59:42 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:42 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:42.598035 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:42.592445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:35919)
0928 21:59:42.592618 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:42.592636 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:42.592720 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:42.592857 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:42.592858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:42.593025 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:42.593077 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:42.593078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.593836 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.593837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:42.594398 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:42.594400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.594838 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.594839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:42.594896 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:42.595594 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:42.595600 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:42.597927 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
W20260928 21:59:42.598116 11893 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:42.592529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:35919 (local address 127.2.165.126:35057)
0928 21:59:42.592583 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:42.592587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:42.592689 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:59:42.592769 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:42.592769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:42.592780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:42.592835 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:42.593195 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.593721 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.594514 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.594731 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.594801 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:42.598038 (+ 3237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:35919: BlockingRecv error: recv got EOF from 127.2.165.66:35919 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 21:59:42 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:42 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:42.738754 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:42.733723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:47975)
0928 21:59:42.733920 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:42.733932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:42.734026 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:42.734212 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:42.734213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:42.734389 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:42.734463 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:42.734466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.735207 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.735208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:42.735764 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:42.735765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.736137 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.736138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:42.736200 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:42.736709 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:42.736714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:42.738684 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":119,"threads_started":1}
W20260928 21:59:42.738924 11893 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:42.733875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:47975 (local address 127.2.165.126:35057)
0928 21:59:42.733949 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:42.733952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:42.734054 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:59:42.734109 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:42.734109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:42.734123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:42.734179 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:42.734538 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.735111 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.735853 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:42.736051 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:42.736138 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:42.738824 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:47975: BlockingRecv error: recv got EOF from 127.2.165.67:47975 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 21:59:43 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:43 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:43.486697 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:43.475209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:35345)
0928 21:59:43.475433 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:43.475446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:43.475525 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:43.475672 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:43.475673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:43.475842 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:43.475892 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:43.475894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.476638 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.476639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:43.477285 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:43.477288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.477690 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.477691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:43.477760 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:43.484074 (+ 6314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:43.484081 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:43.486631 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":136,"threads_started":1}
W20260928 21:59:43.486908 11897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:43.475334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:35345 (local address 127.2.165.126:35057)
0928 21:59:43.475449 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:43.475451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:43.475527 (+ 76us) server_negotiation.cc:407] Connection header received
0928 21:59:43.475575 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:43.475576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:43.475587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:43.475649 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:43.475995 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.476542 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.477374 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.477608 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.477669 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:43.486796 (+ 9127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:35345: BlockingRecv error: recv got EOF from 127.2.165.65:35345 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
Sep 28 21:59:43 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:43 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:43.603869 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:43.598646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:52145)
0928 21:59:43.598997 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:43.599017 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:43.599110 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:43.599294 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:43.599294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:43.599485 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:43.599535 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:43.599537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.600205 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.600206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:43.600781 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:43.600783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.601177 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.601178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:43.601235 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:43.601860 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:43.601865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:43.603808 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
W20260928 21:59:43.604104 11897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:43.598796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:52145 (local address 127.2.165.126:35057)
0928 21:59:43.598971 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:43.598974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:43.599090 (+ 116us) server_negotiation.cc:407] Connection header received
0928 21:59:43.599211 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:43.599212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:43.599223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:43.599289 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:43.599622 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.600120 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.600883 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.601103 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.601179 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:43.604025 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:52145: BlockingRecv error: recv got EOF from 127.2.165.66:52145 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 28 21:59:43 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:43 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:43.744778 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:43.739329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:46921)
0928 21:59:43.739493 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:43.739507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:43.739577 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:43.739790 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:43.739791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:43.739966 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:43.740024 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:43.740026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.740797 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.740799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:43.741338 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:43.741340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.741664 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.741665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:43.741748 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:43.742435 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:43.742440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:43.744640 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
W20260928 21:59:43.744956 11897 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:43.739478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:46921 (local address 127.2.165.126:35057)
0928 21:59:43.739560 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:43.739562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:43.739572 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:43.739659 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:43.739660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:43.739672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:43.739734 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:43.740110 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.740630 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.741415 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:43.741597 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:43.741651 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:43.744879 (+ 3228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:46921: BlockingRecv error: recv got EOF from 127.2.165.67:46921 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 21:59:44 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:44 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:44.492575 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:44.487317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:46113)
0928 21:59:44.487486 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:44.487500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:44.487586 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:44.487765 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:44.487766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:44.488093 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:44.488152 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:44.488154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.488921 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.488922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:44.489505 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:44.489507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.489905 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.489906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:44.489966 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:44.490553 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:44.490558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:44.492468 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
W20260928 21:59:44.492640 11901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:44.487432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:46113 (local address 127.2.165.126:35057)
0928 21:59:44.487573 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:44.487579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:44.487591 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:59:44.487662 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:44.487664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:44.487678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:44.487757 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:44.488239 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.488835 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.489605 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.489814 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.489903 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:44.492565 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:46113: BlockingRecv error: recv got EOF from 127.2.165.65:46113 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
Sep 28 21:59:44 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:44 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:44.609833 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:44.604490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:54607)
0928 21:59:44.604740 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:44.604753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:44.604834 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:44.605066 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:44.605067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:44.605238 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:44.605282 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:44.605283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.605994 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.605994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:44.606547 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:44.606550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.606971 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.606972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:44.607023 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:44.607520 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:44.607525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:44.609760 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":145,"threads_started":1}
W20260928 21:59:44.609966 11901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:44.604569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:54607 (local address 127.2.165.126:35057)
0928 21:59:44.604648 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:44.604650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:44.604836 (+ 186us) server_negotiation.cc:407] Connection header received
0928 21:59:44.604899 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:44.604900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:44.604911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:44.604966 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:44.605359 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.605919 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.606666 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.606860 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.606920 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:44.609860 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:54607: BlockingRecv error: recv got EOF from 127.2.165.66:54607 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 21:59:44 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:44 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:44.750536 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:44.745360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:49613)
0928 21:59:44.745524 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:44.745540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:44.745598 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:44.745746 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:44.745747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:44.745942 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:44.745995 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:44.745997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.746666 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.746667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:44.747274 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:44.747277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.747669 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.747670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:44.747730 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:44.748382 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:44.748387 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:44.750455 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":115,"threads_started":1}
W20260928 21:59:44.750691 11901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:44.745452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:49613 (local address 127.2.165.126:35057)
0928 21:59:44.745538 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:44.745540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:44.745586 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:59:44.745658 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:44.745658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:44.745670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:44.745726 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:44.746065 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.746566 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.747384 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:44.747571 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:44.747639 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:44.750591 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:49613: BlockingRecv error: recv got EOF from 127.2.165.67:49613 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 21:59:45 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:45 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:45.498276 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:45.493180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:50027)
0928 21:59:45.493368 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:45.493380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:45.493441 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:45.493618 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:45.493619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:45.493790 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:45.493838 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:45.493840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.494627 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.494629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:45.495209 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:45.495211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.495579 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.495580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:45.495631 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:45.496121 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:45.496126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:45.498215 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
W20260928 21:59:45.498466 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:45.493300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:50027 (local address 127.2.165.126:35057)
0928 21:59:45.493435 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:45.493437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:45.493447 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:45.493535 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:45.493536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:45.493547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:45.493607 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:45.493967 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.494507 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.495291 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.495481 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.495535 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:45.498378 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:50027: BlockingRecv error: recv got EOF from 127.2.165.65:50027 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":91,"threads_started":1}
Sep 28 21:59:45 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:45 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:45.615504 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:45.610378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:33277)
0928 21:59:45.610571 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:45.610583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:45.610663 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:45.610795 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:45.610796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:45.610967 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:45.611095 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:45.611097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.611780 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.611781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:45.612380 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:45.612382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.612780 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.612781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:45.612832 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:45.613415 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:45.613420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:45.615429 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
W20260928 21:59:45.615679 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:45.610471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:33277 (local address 127.2.165.126:35057)
0928 21:59:45.610526 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:45.610528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:45.610662 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:59:45.610724 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:45.610725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:45.610739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:45.610790 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:45.611181 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.611693 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.612468 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.612700 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.612788 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:45.615570 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:33277: BlockingRecv error: recv got EOF from 127.2.165.66:33277 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 21:59:45 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:45 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:45.756392 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:45.751255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:33691)
0928 21:59:45.751458 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:45.751483 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:45.751563 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:45.751837 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:45.751838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:45.752027 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:45.752074 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:45.752076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.752795 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.752796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:45.753488 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:45.753490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.753953 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.753954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:45.754031 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:45.754537 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:45.754541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:45.756331 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
W20260928 21:59:45.756625 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:45.751412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:33691 (local address 127.2.165.126:35057)
0928 21:59:45.751512 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:45.751514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:45.751544 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:59:45.751706 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:45.751707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:45.751720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:45.751784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:45.752190 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.752720 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.753585 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:45.753881 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:45.753956 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:45.756514 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:33691: BlockingRecv error: recv got EOF from 127.2.165.67:33691 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 21:59:46 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:46 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:46.504213 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:46.498882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:36533)
0928 21:59:46.499027 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:46.499038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:46.499097 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:46.499240 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:46.499240 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:46.499420 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:46.499469 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:46.499471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.500232 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.500233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:46.500874 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:46.500876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.501254 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.501255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:46.501298 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:46.501875 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:46.501880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:46.504150 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260928 21:59:46.504415 11909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:46.498956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:36533 (local address 127.2.165.126:35057)
0928 21:59:46.499120 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:46.499122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:46.499131 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:46.499156 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:46.499156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:46.499167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:46.499238 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:46.499584 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.500109 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.500959 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.501155 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.501240 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:46.504335 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:36533: BlockingRecv error: recv got EOF from 127.2.165.65:36533 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
Sep 28 21:59:46 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:46 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:46.621373 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:46.616056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:50159)
0928 21:59:46.616216 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:46.616230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:46.616304 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:46.616541 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:46.616542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:46.616833 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:46.616883 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:46.616885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.617569 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.617571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:46.618167 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:46.618169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.618747 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.618748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:46.618813 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:46.619394 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:46.619399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:46.621313 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
W20260928 21:59:46.621485 11909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:46.616273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:50159 (local address 127.2.165.126:35057)
0928 21:59:46.616355 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:46.616357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:46.616370 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:59:46.616405 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:46.616406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:46.616417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:46.616473 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:46.616960 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.617469 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.618301 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.618650 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.618721 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:46.621412 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:50159: BlockingRecv error: recv got EOF from 127.2.165.66:50159 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 21:59:46 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:46 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:46.761951 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:46.756914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:40455)
0928 21:59:46.757069 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:46.757084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:46.757186 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:46.757398 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:46.757399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:46.757584 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:46.757629 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:46.757631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.758328 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.758330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:46.758874 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:46.758875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.759259 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.759260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:46.759326 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:46.759873 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:46.759877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:46.761828 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260928 21:59:46.761989 11909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:46.757015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:40455 (local address 127.2.165.126:35057)
0928 21:59:46.757094 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:46.757096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:46.757171 (+ 75us) server_negotiation.cc:407] Connection header received
0928 21:59:46.757303 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:46.757304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:46.757319 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:46.757391 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:46.757725 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.758228 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.758978 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:46.759182 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:46.759260 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:46.761924 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:40455: BlockingRecv error: recv got EOF from 127.2.165.67:40455 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 21:59:47 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:47 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:47.509753 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:47.504837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:37721)
0928 21:59:47.504999 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:47.505012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:47.505147 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:47.505414 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:47.505416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:47.505593 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:47.505641 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:47.505643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.506361 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.506362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:47.506911 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:47.506913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.507300 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.507301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:47.507350 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:47.507827 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:47.507832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:47.509682 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
W20260928 21:59:47.509895 11913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:47.504987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:37721 (local address 127.2.165.126:35057)
0928 21:59:47.505121 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:47.505124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:47.505135 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:47.505298 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:47.505299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:47.505309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:47.505369 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:47.505728 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.506288 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.506997 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.507221 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.507280 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:47.509784 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:37721: BlockingRecv error: recv got EOF from 127.2.165.65:37721 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":52,"threads_started":1}
Sep 28 21:59:47 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:47 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:47.626914 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:47.621986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:55337)
0928 21:59:47.622139 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:47.622153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:47.622223 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:47.622488 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:47.622490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:47.622784 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:47.622850 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:47.622852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.623594 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.623595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:47.624176 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:47.624178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.624558 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.624558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:47.624603 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:47.625080 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:47.625085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:47.626854 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
W20260928 21:59:47.627141 11913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:47.622082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:55337 (local address 127.2.165.126:35057)
0928 21:59:47.622143 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:47.622146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:47.622301 (+ 155us) server_negotiation.cc:407] Connection header received
0928 21:59:47.622336 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:47.622337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:47.622348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:47.622426 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:47.622948 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.623513 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.624276 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.624474 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.624544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:47.627076 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:55337: BlockingRecv error: recv got EOF from 127.2.165.66:55337 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 21:59:47 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:47 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.67@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:47.767895 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:47.762645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.67:59671)
0928 21:59:47.762792 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:47.762812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:47.762871 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:47.763191 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:47.763192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:47.763395 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:47.763444 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:47.763446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.764129 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.764130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:47.764798 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:47.764800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.765297 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.765299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:47.765379 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:47.765822 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:47.765826 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:47.767835 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
W20260928 21:59:47.768033 11913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:47.762762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:59671 (local address 127.2.165.126:35057)
0928 21:59:47.762822 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:47.762824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:47.762857 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:59:47.762930 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:47.762931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:47.762942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:47.763005 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:47.763532 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.764047 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.764898 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:47.765222 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:47.765286 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:47.767952 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.67:59671: BlockingRecv error: recv got EOF from 127.2.165.67:59671 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 21:59:48 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:48 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.65@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:48.523303 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:48.510326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.65:37291)
0928 21:59:48.510457 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:48.510476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:48.510527 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:48.510775 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:48.510775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:48.510939 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:48.510988 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:48.510990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.511857 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.511858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:48.512403 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:48.512404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.512801 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.512802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:48.512846 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:48.520949 (+ 8103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:48.520957 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:48.523207 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
W20260928 21:59:48.523386 11917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:48.510452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:37291 (local address 127.2.165.126:35057)
0928 21:59:48.510614 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:48.510616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:48.510625 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:48.510650 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:48.510650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:48.510660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:48.510762 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:48.511095 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.511762 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.512489 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.512728 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.512788 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:48.523312 (+ 10524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.65:37291: BlockingRecv error: recv got EOF from 127.2.165.65:37291 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
Sep 28 21:59:48 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:59:48 dist-test-slave-3t36 krb5kdc[10859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.165.66@KRBTEST.COM for oryx/127.2.165.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:59:48.632504 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:48.627450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35057 (local address 127.2.165.66:44957)
0928 21:59:48.627652 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:48.627664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:48.627743 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:48.627918 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:48.627919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:48.628115 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:48.628161 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:48.628163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.628918 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.628919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:48.629512 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:48.629514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.629906 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.629907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:48.629967 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:48.630411 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:48.630416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:48.632433 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.165.126:35057: Server oryx/127.2.165.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
W20260928 21:59:48.632661 11917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:48.627554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:44957 (local address 127.2.165.126:35057)
0928 21:59:48.627630 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:48.627632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:48.627737 (+ 105us) server_negotiation.cc:407] Connection header received
0928 21:59:48.627812 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:48.627813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:48.627832 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:48.627914 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:48.628248 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.628834 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.629613 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.629824 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.629897 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:48.632557 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.165.66:44957: BlockingRecv error: recv got EOF from 127.2.165.66:44957 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:59:48.679598 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 11442
I20260928 21:59:48.684724 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 11594
I20260928 21:59:48.690279 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 11727
I20260928 21:59:48.695011 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 11372
2026-09-28T21:59:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17492 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T21:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:48Z Disabled control of system clock
I20260928 21:59:48.724771 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:39145
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:59:48.807807 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:48.808032 11922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:48.808053 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:48.809322 11922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:48.809366 11922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:48.809379 11922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:48.809392 11922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:48.809404 11922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:48.810735 11922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:39145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:48.810940 11922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:48.811203 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:48.816565 11928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:48.816563 11927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:48.816763 11922 server_base.cc:1034] running on GCE node
W20260928 21:59:48.816560 11930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:48.817417 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:48.817781 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:48.818995 11922 hybrid_clock.cc:648] HybridClock initialized: now 1790632788818979 us; error 40 us; skew 500 ppm
I20260928 21:59:48.820538 11922 webserver.cc:460] Webserver started at http://127.2.165.126:39913/ using document root <none> and password file <none>
I20260928 21:59:48.820914 11922 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:48.820998 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:48.821170 11922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:48.822475 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "6e83c026763b47858c12c3e1b518b993"
format_stamp: "Formatted at 2026-09-28 21:59:48 on dist-test-slave-3t36"
server_key: "098003e45fef90239f57195caa76518f"
server_key_iv: "77e6e5b23f788717bae5d0cbc1d3a561"
server_key_version: "encryptionkey@0"
I20260928 21:59:48.822890 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "6e83c026763b47858c12c3e1b518b993"
format_stamp: "Formatted at 2026-09-28 21:59:48 on dist-test-slave-3t36"
server_key: "098003e45fef90239f57195caa76518f"
server_key_iv: "77e6e5b23f788717bae5d0cbc1d3a561"
server_key_version: "encryptionkey@0"
I20260928 21:59:48.824338 11922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:48.825367 11936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:48.825565 11922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:59:48.825690 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "6e83c026763b47858c12c3e1b518b993"
format_stamp: "Formatted at 2026-09-28 21:59:48 on dist-test-slave-3t36"
server_key: "098003e45fef90239f57195caa76518f"
server_key_iv: "77e6e5b23f788717bae5d0cbc1d3a561"
server_key_version: "encryptionkey@0"
I20260928 21:59:48.825774 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:48.848436 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:48.848869 11922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:48.849045 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:48.853681 11922 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:39145
I20260928 21:59:48.853709 11988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:39145 every 8 connection(s)
I20260928 21:59:48.854101 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:48.854817 11989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:48.857574 11989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993: Bootstrap starting.
I20260928 21:59:48.858186 11989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:48.858494 11989 log.cc:824] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:48.859302 11989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993: No bootstrap required, opened a new log
I20260928 21:59:48.859632 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 11922
I20260928 21:59:48.859740 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:48.859912 2709 external_mini_cluster.cc:1428] Setting key 23aa29ce75c5ba09b57d3376805c7ba5
I20260928 21:59:48.861544 11989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e83c026763b47858c12c3e1b518b993" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39145 } }
I20260928 21:59:48.861781 11989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:48.861843 11989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e83c026763b47858c12c3e1b518b993, State: Initialized, Role: FOLLOWER
I20260928 21:59:48.861981 11989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e83c026763b47858c12c3e1b518b993" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39145 } }
I20260928 21:59:48.862154 11989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:48.862205 11989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:48.862267 11989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:48.863029 11989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e83c026763b47858c12c3e1b518b993" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39145 } }
I20260928 21:59:48.863147 11989 leader_election.cc:302] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e83c026763b47858c12c3e1b518b993; no voters:
I20260928 21:59:48.863296 11989 leader_election.cc:288] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:48.863364 11994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:48.863524 11989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:48.863564 11994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [term 1 LEADER]: Becoming Leader. State: Replica: 6e83c026763b47858c12c3e1b518b993, State: Running, Role: LEADER
I20260928 21:59:48.863682 11994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e83c026763b47858c12c3e1b518b993" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39145 } }
I20260928 21:59:48.864073 11994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e83c026763b47858c12c3e1b518b993. Latest consensus state: current_term: 1 leader_uuid: "6e83c026763b47858c12c3e1b518b993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e83c026763b47858c12c3e1b518b993" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39145 } } }
I20260928 21:59:48.864341 11994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:48.864262 11995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e83c026763b47858c12c3e1b518b993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e83c026763b47858c12c3e1b518b993" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39145 } } }
I20260928 21:59:48.864802 12003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:48.864806 11995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:48.865442 12003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:48.867430 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:48.860494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.2.165.126:39145)
0928 21:59:48.860764 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:48.860773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:48.860801 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:59:48.861470 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:48.861478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:48.861543 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:48.861677 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:48.861951 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.863667 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.864910 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:48.865627 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:48.865754 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:48.866654 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:48.866659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:48.866660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:48.866893 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:48.866931 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:48.866945 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:48.866982 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:48.867148 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 21:59:48.867244 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260928 21:59:48.867782 12003 catalog_manager.cc:1347] Generated new cluster ID: 9ce59dc033e1469cb9490dd58f599b5b
I20260928 21:59:48.867846 12003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:48.872504 12003 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:48.873241 12003 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:48.878527 12003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e83c026763b47858c12c3e1b518b993: Generated new TSK 0
I20260928 21:59:48.878751 12003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:59:48.881381 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39145
--builtin_ntp_servers=127.2.165.84:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:59:48.967191 12013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:48.967427 12013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:48.967450 12013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:48.968796 12013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:48.968839 12013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:48.968868 12013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:48.970283 12013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:48.970574 12013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:48.970904 12013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:48.976245 12021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:48.976313 12018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:48.976313 12019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:48.976737 12013 server_base.cc:1034] running on GCE node
I20260928 21:59:48.976961 12013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:48.977236 12013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:48.978447 12013 hybrid_clock.cc:648] HybridClock initialized: now 1790632788978426 us; error 33 us; skew 500 ppm
I20260928 21:59:48.979820 12013 webserver.cc:460] Webserver started at http://127.2.165.65:36297/ using document root <none> and password file <none>
I20260928 21:59:48.980073 12013 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:48.980144 12013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:48.980286 12013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:48.981451 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "34ba01f0870e40dfaec4e31292b6265b"
format_stamp: "Formatted at 2026-09-28 21:59:48 on dist-test-slave-3t36"
server_key: "10afef3edabdc0a9fb3b635f1cef6558"
server_key_iv: "ec290255978ec14c499f1a2bf47593a4"
server_key_version: "encryptionkey@0"
I20260928 21:59:48.981832 12013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "34ba01f0870e40dfaec4e31292b6265b"
format_stamp: "Formatted at 2026-09-28 21:59:48 on dist-test-slave-3t36"
server_key: "10afef3edabdc0a9fb3b635f1cef6558"
server_key_iv: "ec290255978ec14c499f1a2bf47593a4"
server_key_version: "encryptionkey@0"
I20260928 21:59:48.983333 12013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:59:48.984211 12027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:48.984380 12013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:48.984581 12013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "34ba01f0870e40dfaec4e31292b6265b"
format_stamp: "Formatted at 2026-09-28 21:59:48 on dist-test-slave-3t36"
server_key: "10afef3edabdc0a9fb3b635f1cef6558"
server_key_iv: "ec290255978ec14c499f1a2bf47593a4"
server_key_version: "encryptionkey@0"
I20260928 21:59:48.984724 12013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:49.019286 12013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:49.019639 12013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:49.019806 12013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:49.020066 12013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:49.020453 12013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:49.020488 12013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.020512 12013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:49.020524 12013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.026788 12013 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:33961
I20260928 21:59:49.026852 12140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:33961 every 8 connection(s)
I20260928 21:59:49.027240 12013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:49.032534 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.028022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:48355 (local address 127.2.165.126:39145)
0928 21:59:49.028081 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.028083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.028358 (+ 275us) server_negotiation.cc:407] Connection header received
0928 21:59:49.028855 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.028856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.028866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.028922 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.029596 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.030116 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.031208 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.031392 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.031452 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:49.032284 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:49.032286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:49.032286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:49.032290 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:49.032305 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:49.032309 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:49.032356 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:49.032450 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:59:49.032471 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:59:49.032851 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.028005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39145 (local address 127.2.165.65:48355)
0928 21:59:49.028203 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:49.028239 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:49.028763 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:49.029018 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:49.029021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:49.029029 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:49.029526 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:49.029530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.030284 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.030285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:49.031116 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:49.031118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.031466 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.031466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:49.031576 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:49.032135 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:49.032136 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:49.032168 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:49.032192 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:49.032380 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:49.032384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:49.032386 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:59:49.032440 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:59:49.032527 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260928 21:59:49.033489 12141 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39145
I20260928 21:59:49.033604 12141 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:49.033800 12141 heartbeater.cc:511] Master 127.2.165.126:39145 requested a full tablet report, sending...
I20260928 21:59:49.034379 11953 ts_manager.cc:194] Registered new tserver with Master: 34ba01f0870e40dfaec4e31292b6265b (127.2.165.65:33961)
I20260928 21:59:49.034963 11953 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.65:48355
I20260928 21:59:49.037184 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12013
I20260928 21:59:49.037320 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:49.037493 2709 external_mini_cluster.cc:1428] Setting key 3a85c514f097ea83d111497536c54f72
I20260928 21:59:49.038542 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39145
--builtin_ntp_servers=127.2.165.84:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:59:49.121722 12144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:49.121955 12144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:49.121979 12144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:49.123399 12144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:49.123449 12144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:49.123476 12144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:49.125010 12144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:49.125272 12144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:49.125595 12144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:49.131134 12152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.131114 12150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.131114 12149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.131153 12144 server_base.cc:1034] running on GCE node
I20260928 21:59:49.131664 12144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:49.131886 12144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:49.133030 12144 hybrid_clock.cc:648] HybridClock initialized: now 1790632789133018 us; error 31 us; skew 500 ppm
I20260928 21:59:49.134404 12144 webserver.cc:460] Webserver started at http://127.2.165.66:36489/ using document root <none> and password file <none>
I20260928 21:59:49.134651 12144 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:49.134699 12144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:49.134889 12144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:49.135993 12144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "217252876f0d49e98a313ef09fdb14cb"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "dcded6b7db49a0952087d304bed45be2"
server_key_iv: "7460195877a318598ef916690efb0284"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.136489 12144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "217252876f0d49e98a313ef09fdb14cb"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "dcded6b7db49a0952087d304bed45be2"
server_key_iv: "7460195877a318598ef916690efb0284"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.138254 12144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:49.139074 12158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.139254 12144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:49.139357 12144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "217252876f0d49e98a313ef09fdb14cb"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "dcded6b7db49a0952087d304bed45be2"
server_key_iv: "7460195877a318598ef916690efb0284"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.139456 12144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:49.149942 12144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:49.150305 12144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:49.150481 12144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:49.150717 12144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:49.151064 12144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:49.151119 12144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.151162 12144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:49.151188 12144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.157205 12144 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:44281
I20260928 21:59:49.157243 12271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:44281 every 8 connection(s)
I20260928 21:59:49.157613 12144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:49.163249 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.158383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:60323 (local address 127.2.165.126:39145)
0928 21:59:49.158452 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.158455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.158716 (+ 261us) server_negotiation.cc:407] Connection header received
0928 21:59:49.159278 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.159279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.159291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.159353 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.160045 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.160562 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.161639 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.161830 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.161911 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:49.162893 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:49.162894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:49.162895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:49.162899 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:49.162916 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:49.162919 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:49.162986 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:49.163161 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 21:59:49.163188 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:59:49.163373 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.158383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39145 (local address 127.2.165.66:60323)
0928 21:59:49.158586 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:49.158623 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:49.159130 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:49.159432 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:49.159436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:49.159444 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:49.159948 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:49.159952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.160647 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.160648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:49.161540 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:49.161542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.161957 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.161958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:49.162088 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:49.162739 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:49.162739 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:49.162770 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:49.162794 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:49.162999 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:49.163004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:49.163006 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:59:49.163064 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:59:49.163153 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260928 21:59:49.163712 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12144
I20260928 21:59:49.163792 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:49.163978 2709 external_mini_cluster.cc:1428] Setting key f6f4fc9df1638abf0aadf92e94fe71c8
I20260928 21:59:49.164476 12272 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39145
I20260928 21:59:49.164777 12272 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:49.165032 12272 heartbeater.cc:511] Master 127.2.165.126:39145 requested a full tablet report, sending...
I20260928 21:59:49.165238 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39145
--builtin_ntp_servers=127.2.165.84:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:59:49.165500 11952 ts_manager.cc:194] Registered new tserver with Master: 217252876f0d49e98a313ef09fdb14cb (127.2.165.66:44281)
I20260928 21:59:49.165910 11952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.66:60323
W20260928 21:59:49.248215 12275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:49.248526 12275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:49.248575 12275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:49.249846 12275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:49.249893 12275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:49.249962 12275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:49.251351 12275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:49.251662 12275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:49.251991 12275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:49.257140 12283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.257261 12281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.257470 12280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.257738 12275 server_base.cc:1034] running on GCE node
I20260928 21:59:49.257969 12275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:49.258183 12275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:49.259352 12275 hybrid_clock.cc:648] HybridClock initialized: now 1790632789259339 us; error 29 us; skew 500 ppm
I20260928 21:59:49.260761 12275 webserver.cc:460] Webserver started at http://127.2.165.67:36681/ using document root <none> and password file <none>
I20260928 21:59:49.261020 12275 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:49.261070 12275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:49.261236 12275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:49.262382 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "edb20951f42a40ae917b1fb5aca42af6"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "bfbd66cfebdbb310d8a0bc0fa8f98174"
server_key_iv: "063398189557c7cf5bc5fc8ba2ee1d82"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.262769 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "edb20951f42a40ae917b1fb5aca42af6"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "bfbd66cfebdbb310d8a0bc0fa8f98174"
server_key_iv: "063398189557c7cf5bc5fc8ba2ee1d82"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.264374 12275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:49.265221 12289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.265400 12275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:59:49.265498 12275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "edb20951f42a40ae917b1fb5aca42af6"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "bfbd66cfebdbb310d8a0bc0fa8f98174"
server_key_iv: "063398189557c7cf5bc5fc8ba2ee1d82"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.265579 12275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:49.279110 12275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:49.279460 12275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:49.279605 12275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:49.279857 12275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:49.280223 12275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:49.280283 12275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.280328 12275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:49.280354 12275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.286289 12275 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:44511
I20260928 21:59:49.286334 12402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:44511 every 8 connection(s)
I20260928 21:59:49.286669 12275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:49.290879 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12275
I20260928 21:59:49.290980 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:49.291165 2709 external_mini_cluster.cc:1428] Setting key 95974ce5c1f1993af28a962582d3ab5e
I20260928 21:59:49.292733 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.287378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:50713 (local address 127.2.165.126:39145)
0928 21:59:49.287450 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.287453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.287779 (+ 326us) server_negotiation.cc:407] Connection header received
0928 21:59:49.288387 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.288389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.288399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.288460 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.289338 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.289852 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.290887 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.291209 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.291285 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:49.292418 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:49.292420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:49.292420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:49.292425 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:49.292442 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:49.292451 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:49.292510 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:49.292584 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:59:49.292612 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:59:49.292980 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.287376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39145 (local address 127.2.165.67:50713)
0928 21:59:49.287602 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:49.287662 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:49.288266 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:49.288597 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:49.288600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:49.288609 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:49.289231 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:49.289237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.289952 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.289953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:49.290784 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:49.290786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:49.291370 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:49.291371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:49.291539 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:49.292258 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:49.292258 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:49.292310 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:49.292340 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:49.292511 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:49.292517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:49.292521 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:49.292590 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:59:49.292723 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
I20260928 21:59:49.293484 12403 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39145
I20260928 21:59:49.293562 12403 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:49.293761 12403 heartbeater.cc:511] Master 127.2.165.126:39145 requested a full tablet report, sending...
I20260928 21:59:49.294137 11952 ts_manager.cc:194] Registered new tserver with Master: edb20951f42a40ae917b1fb5aca42af6 (127.2.165.67:44511)
I20260928 21:59:49.294523 11952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.67:50713
I20260928 21:59:49.303681 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:59:49.305783 11993 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:49.305310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.2.165.126:39145)
0928 21:59:49.305389 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.305390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.305400 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:59:49.305439 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.305440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.305450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.305502 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.305631 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55800: BlockingRecv error: recv got EOF from 127.0.0.1:55800 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:59:49.305857 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12013
I20260928 21:59:49.310357 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12144
I20260928 21:59:49.315202 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12275
I20260928 21:59:49.319577 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 11922
2026-09-28T21:59:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (622 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T21:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:49Z Disabled control of system clock
I20260928 21:59:49.346815 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41529
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:41529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:59:49.428584 12414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:49.428843 12414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:49.428865 12414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:49.430022 12414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:49.430063 12414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:49.430075 12414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:49.430088 12414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:49.430100 12414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:49.431596 12414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:41529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:49.431825 12414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:49.432075 12414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:49.437101 12419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.437132 12422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.437100 12420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.437475 12414 server_base.cc:1034] running on GCE node
I20260928 21:59:49.437728 12414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:49.438079 12414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:49.439280 12414 hybrid_clock.cc:648] HybridClock initialized: now 1790632789439255 us; error 56 us; skew 500 ppm
I20260928 21:59:49.440749 12414 webserver.cc:460] Webserver started at http://127.2.165.126:43517/ using document root <none> and password file <none>
I20260928 21:59:49.441049 12414 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:49.441120 12414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:49.441298 12414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:49.442440 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "dcc3c3ccf99c49b7b09181f52d6aa593"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "a51c5d4f57937020775522ea9b28f918"
server_key_iv: "a18d6739d19f6ee0febf526df52f7ad0"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.442827 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "dcc3c3ccf99c49b7b09181f52d6aa593"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "a51c5d4f57937020775522ea9b28f918"
server_key_iv: "a18d6739d19f6ee0febf526df52f7ad0"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.444295 12414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:49.445467 12428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.445720 12414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:49.445849 12414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "dcc3c3ccf99c49b7b09181f52d6aa593"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "a51c5d4f57937020775522ea9b28f918"
server_key_iv: "a18d6739d19f6ee0febf526df52f7ad0"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.445943 12414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:49.449538 12414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:49.449862 12414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:49.450014 12414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:49.454043 12414 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:41529
I20260928 21:59:49.454087 12480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:41529 every 8 connection(s)
I20260928 21:59:49.454463 12414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:49.455180 12481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:49.458160 12481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593: Bootstrap starting.
I20260928 21:59:49.458719 12481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:49.458958 12481 log.cc:824] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:49.459661 12481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593: No bootstrap required, opened a new log
I20260928 21:59:49.461433 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12414
I20260928 21:59:49.461406 12481 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41529 } }
I20260928 21:59:49.461544 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:49.461534 12481 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:49.461592 12481 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc3c3ccf99c49b7b09181f52d6aa593, State: Initialized, Role: FOLLOWER
I20260928 21:59:49.461760 2709 external_mini_cluster.cc:1428] Setting key 8f3677657db95a0a5d7f08c0b102d332
I20260928 21:59:49.461750 12481 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41529 } }
I20260928 21:59:49.461843 12481 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:49.461884 12481 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:49.461942 12481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:49.462714 12481 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41529 } }
I20260928 21:59:49.462838 12481 leader_election.cc:302] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcc3c3ccf99c49b7b09181f52d6aa593; no voters:
I20260928 21:59:49.463016 12481 leader_election.cc:288] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:49.463061 12486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:49.463274 12486 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [term 1 LEADER]: Becoming Leader. State: Replica: dcc3c3ccf99c49b7b09181f52d6aa593, State: Running, Role: LEADER
I20260928 21:59:49.463443 12486 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41529 } }
I20260928 21:59:49.463690 12481 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:49.463807 12486 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcc3c3ccf99c49b7b09181f52d6aa593. Latest consensus state: current_term: 1 leader_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41529 } } }
I20260928 21:59:49.463889 12486 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:49.464085 12490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:49.464221 12488 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc3c3ccf99c49b7b09181f52d6aa593" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41529 } } }
I20260928 21:59:49.464365 12488 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:49.464531 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.462362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.2.165.126:41529)
0928 21:59:49.462583 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.462591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.462618 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:59:49.462711 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.462717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.462751 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.462936 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.463791 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:49.463795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:49.463796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:49.463801 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:49.463851 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:49.463868 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:49.464220 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:49.464255 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:59:49.464325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260928 21:59:49.464645 12490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:49.466411 12490 catalog_manager.cc:1347] Generated new cluster ID: 8f95de7d8bb74143abaf0f3d317aeb80
I20260928 21:59:49.466491 12490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:59:49.476940 12490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:49.477146 12490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:49.484781 12490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcc3c3ccf99c49b7b09181f52d6aa593: Generated new TSK 0
I20260928 21:59:49.485033 12490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:59:49.493623 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41529
--builtin_ntp_servers=127.2.165.84:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:59:49.576246 12505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:49.576465 12505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:49.576489 12505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:49.577739 12505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:49.577786 12505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:49.577814 12505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:49.579238 12505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:49.579511 12505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:49.579834 12505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:49.585130 12510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.585304 12505 server_base.cc:1034] running on GCE node
W20260928 21:59:49.585162 12513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.585131 12511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.585592 12505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:49.585856 12505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:49.587074 12505 hybrid_clock.cc:648] HybridClock initialized: now 1790632789587051 us; error 44 us; skew 500 ppm
I20260928 21:59:49.588498 12505 webserver.cc:460] Webserver started at http://127.2.165.65:38915/ using document root <none> and password file <none>
I20260928 21:59:49.588829 12505 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:49.588899 12505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:49.589057 12505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:49.590165 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "6bcb5da9baf14bf5965e924e8e16468f"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "d1fa2790e2158c78ef8c47d3d857cd2b"
server_key_iv: "f8e30bdc42cfdb649ca48dd5878c843a"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.590531 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "6bcb5da9baf14bf5965e924e8e16468f"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "d1fa2790e2158c78ef8c47d3d857cd2b"
server_key_iv: "f8e30bdc42cfdb649ca48dd5878c843a"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.592026 12505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:49.592976 12519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.593199 12505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:49.593318 12505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "6bcb5da9baf14bf5965e924e8e16468f"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "d1fa2790e2158c78ef8c47d3d857cd2b"
server_key_iv: "f8e30bdc42cfdb649ca48dd5878c843a"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.593408 12505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:49.596868 12505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:49.597158 12505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:49.597308 12505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:49.597527 12505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:49.597847 12505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:49.597901 12505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.597944 12505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:49.597971 12505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.603864 12505 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:45389
I20260928 21:59:49.603935 12632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:45389 every 8 connection(s)
I20260928 21:59:49.604318 12505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:49.607546 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.605137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:37699 (local address 127.2.165.126:41529)
0928 21:59:49.605181 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.605183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.605463 (+ 280us) server_negotiation.cc:407] Connection header received
0928 21:59:49.605995 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.605996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.606004 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.606061 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.607260 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:49.607261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:49.607262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:49.607266 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:49.607284 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:49.607289 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:49.607361 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:49.607456 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 21:59:49.607474 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:59:49.607795 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.605123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:41529 (local address 127.2.165.65:37699)
0928 21:59:49.605334 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:49.605368 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:49.605884 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:49.606147 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:49.606151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:49.606163 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:49.607081 (+ 918us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:49.607082 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:49.607112 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:49.607142 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:49.607358 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:49.607365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:49.607368 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:49.607468 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:59:49.607534 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260928 21:59:49.608316 12633 heartbeater.cc:347] Connected to a master server at 127.2.165.126:41529
I20260928 21:59:49.608428 12633 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:49.608515 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12505
I20260928 21:59:49.608603 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:49.608601 12633 heartbeater.cc:511] Master 127.2.165.126:41529 requested a full tablet report, sending...
I20260928 21:59:49.608846 2709 external_mini_cluster.cc:1428] Setting key fbd00dbac83fa652c5a66df9f27de701
I20260928 21:59:49.609206 12445 ts_manager.cc:194] Registered new tserver with Master: 6bcb5da9baf14bf5965e924e8e16468f (127.2.165.65:45389)
I20260928 21:59:49.609970 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41529
--builtin_ntp_servers=127.2.165.84:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:59:49.693200 12636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:49.693449 12636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:49.693490 12636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:49.694761 12636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:49.694834 12636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:49.694882 12636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:49.696434 12636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:49.696766 12636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:49.697165 12636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:49.702467 12644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.702423 12642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.702423 12641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.702548 12636 server_base.cc:1034] running on GCE node
I20260928 21:59:49.702967 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:49.703245 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:49.704440 12636 hybrid_clock.cc:648] HybridClock initialized: now 1790632789704439 us; error 44 us; skew 500 ppm
I20260928 21:59:49.705889 12636 webserver.cc:460] Webserver started at http://127.2.165.66:43913/ using document root <none> and password file <none>
I20260928 21:59:49.706163 12636 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:49.706238 12636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:49.706378 12636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:49.707486 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "d80bfbe70f8446848dc067c4d112b600"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "df7e40d1d1a124b6720fd383cf07d5a4"
server_key_iv: "9b9a21e4ac6111e5cd637b57d53ff249"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.707877 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "d80bfbe70f8446848dc067c4d112b600"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "df7e40d1d1a124b6720fd383cf07d5a4"
server_key_iv: "9b9a21e4ac6111e5cd637b57d53ff249"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.709353 12636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:49.710194 12650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.710403 12636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:49.710517 12636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "d80bfbe70f8446848dc067c4d112b600"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "df7e40d1d1a124b6720fd383cf07d5a4"
server_key_iv: "9b9a21e4ac6111e5cd637b57d53ff249"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.710644 12636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:49.714030 12636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:49.714323 12636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:49.714464 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:49.714686 12636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:49.715049 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:49.715081 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.715109 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:49.715157 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.720781 12636 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:43029
I20260928 21:59:49.720837 12763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:43029 every 8 connection(s)
I20260928 21:59:49.721199 12636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:49.724251 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.721946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:47125 (local address 127.2.165.126:41529)
0928 21:59:49.722010 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.722012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.722239 (+ 227us) server_negotiation.cc:407] Connection header received
0928 21:59:49.722683 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.722684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.722689 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.722754 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.723938 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:49.723939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:49.723940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:49.723945 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:49.723964 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:49.723969 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:49.724035 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:49.724163 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 21:59:49.724180 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:59:49.724442 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.721933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:41529 (local address 127.2.165.66:47125)
0928 21:59:49.722105 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:49.722138 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:49.722590 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:49.722815 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:49.722818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:49.722826 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:49.723749 (+ 923us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:49.723750 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:49.723780 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:49.723811 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:49.724065 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:49.724071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:49.724075 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:49.724159 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:59:49.724224 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260928 21:59:49.724529 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12636
I20260928 21:59:49.724617 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:49.724877 2709 external_mini_cluster.cc:1428] Setting key f5546afbfb8b0e9c5825f9a9e52dff8e
I20260928 21:59:49.724999 12764 heartbeater.cc:347] Connected to a master server at 127.2.165.126:41529
I20260928 21:59:49.725093 12764 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:49.725251 12764 heartbeater.cc:511] Master 127.2.165.126:41529 requested a full tablet report, sending...
I20260928 21:59:49.725664 12444 ts_manager.cc:194] Registered new tserver with Master: d80bfbe70f8446848dc067c4d112b600 (127.2.165.66:43029)
I20260928 21:59:49.725993 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41529
--builtin_ntp_servers=127.2.165.84:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:59:49.809692 12767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:49.809952 12767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:49.809991 12767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:49.811203 12767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:49.811280 12767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:49.811337 12767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:49.812752 12767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:49.813020 12767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:49.813373 12767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:49.818724 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:49.818729 12773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.818813 12767 server_base.cc:1034] running on GCE node
W20260928 21:59:49.818724 12772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:49.819307 12767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:49.819591 12767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:49.820749 12767 hybrid_clock.cc:648] HybridClock initialized: now 1790632789820738 us; error 33 us; skew 500 ppm
I20260928 21:59:49.822265 12767 webserver.cc:460] Webserver started at http://127.2.165.67:45413/ using document root <none> and password file <none>
I20260928 21:59:49.822551 12767 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:49.822623 12767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:49.822785 12767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:49.823899 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "276700dd4ee6457a97533847ea3ed278"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "995bb3b4aaa9a956f74c78cd3993e52c"
server_key_iv: "bb9bd6841993366a375781aa262c1f02"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.824340 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "276700dd4ee6457a97533847ea3ed278"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "995bb3b4aaa9a956f74c78cd3993e52c"
server_key_iv: "bb9bd6841993366a375781aa262c1f02"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.825837 12767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:49.826757 12781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.826969 12767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:49.827132 12767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "276700dd4ee6457a97533847ea3ed278"
format_stamp: "Formatted at 2026-09-28 21:59:49 on dist-test-slave-3t36"
server_key: "995bb3b4aaa9a956f74c78cd3993e52c"
server_key_iv: "bb9bd6841993366a375781aa262c1f02"
server_key_version: "encryptionkey@0"
I20260928 21:59:49.827220 12767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:49.830885 12767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:49.831171 12767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:49.831316 12767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:49.831534 12767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:49.831853 12767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:49.831908 12767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.831947 12767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:49.831976 12767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:49.838140 12767 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:32785
I20260928 21:59:49.838215 12894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:32785 every 8 connection(s)
I20260928 21:59:49.838568 12767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:49.840801 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12767
I20260928 21:59:49.840898 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:49.841090 2709 external_mini_cluster.cc:1428] Setting key b371999e8083837cdd6652e713b9cf06
I20260928 21:59:49.842105 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.839396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:46421 (local address 127.2.165.126:41529)
0928 21:59:49.839484 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.839487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.839698 (+ 211us) server_negotiation.cc:407] Connection header received
0928 21:59:49.840243 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.840245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.840252 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.840313 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.841807 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:49.841808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:49.841808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:59:49.841813 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:49.841832 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:59:49.841836 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:49.841902 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:49.842014 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:59:49.842029 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:59:49.842330 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:49.839362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:41529 (local address 127.2.165.67:46421)
0928 21:59:49.839572 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:49.839607 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:49.840130 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:49.840390 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:49.840395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:49.840411 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:49.841617 (+ 1206us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:59:49.841618 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:49.841664 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:59:49.841719 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:49.841911 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:49.841916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:49.841920 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:49.842011 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:59:49.842077 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 21:59:49.842798 12895 heartbeater.cc:347] Connected to a master server at 127.2.165.126:41529
I20260928 21:59:49.842890 12895 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:49.843010 12895 heartbeater.cc:511] Master 127.2.165.126:41529 requested a full tablet report, sending...
I20260928 21:59:49.843329 12444 ts_manager.cc:194] Registered new tserver with Master: 276700dd4ee6457a97533847ea3ed278 (127.2.165.67:32785)
I20260928 21:59:49.853235 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:59:49.855203 12485 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:59:49.854756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.2.165.126:41529)
0928 21:59:49.854831 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:49.854833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:49.854917 (+ 84us) server_negotiation.cc:407] Connection header received
0928 21:59:49.854949 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:49.854950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:49.854955 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:49.855014 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:49.855120 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55778: BlockingRecv error: recv got EOF from 127.0.0.1:55778 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:59:49.855237 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12505
I20260928 21:59:49.859754 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12636
I20260928 21:59:49.864210 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12767
I20260928 21:59:49.868757 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12414
2026-09-28T21:59:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (548 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:49 dist-test-slave-3t36 krb5kdc[12905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:49 dist-test-slave-3t36 krb5kdc[12905](info): set up 2 sockets
Sep 28 21:59:49 dist-test-slave-3t36 krb5kdc[12905](info): commencing operation
krb5kdc: starting...
W20260928 21:59:50.910409 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:50 dist-test-slave-3t36 krb5kdc[12905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 21:59:51.040867 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45935
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:51.122880 12918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:51.123142 12918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:51.123183 12918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:51.124372 12918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:59:51.124444 12918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:51.124470 12918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:51.124503 12918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:59:51.124524 12918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:59:51.126091 12918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:51.126380 12918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:51.126623 12918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:59:51.131875 12918 server_base.cc:1034] running on GCE node
W20260928 21:59:51.131829 12924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:51.131809 12923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:51.131815 12926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:51.132270 12918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:51.132531 12918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:51.133723 12918 hybrid_clock.cc:648] HybridClock initialized: now 1790632791133702 us; error 49 us; skew 500 ppm
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:51.136262 12918 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 21:59:51.136858 12918 webserver.cc:460] Webserver started at http://127.2.165.126:38383/ using document root <none> and password file <none>
I20260928 21:59:51.137123 12918 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:51.137174 12918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:51.137329 12918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:51.138468 12918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "e324d82d38eb4208b864083fa03623b2"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "d5855c4e7e73c8a1b52221d6bad6c436"
server_key_iv: "1fb007f8ecb21fa23b78ec124c204323"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.138854 12918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "e324d82d38eb4208b864083fa03623b2"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "d5855c4e7e73c8a1b52221d6bad6c436"
server_key_iv: "1fb007f8ecb21fa23b78ec124c204323"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.140285 12918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:51.141211 12933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.141445 12918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:59:51.141564 12918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "e324d82d38eb4208b864083fa03623b2"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "d5855c4e7e73c8a1b52221d6bad6c436"
server_key_iv: "1fb007f8ecb21fa23b78ec124c204323"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.141645 12918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:51.152892 12918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:51.155082 12918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:51.155264 12918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:51.159657 12918 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45935
I20260928 21:59:51.159696 12985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45935 every 8 connection(s)
I20260928 21:59:51.160099 12918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 21:59:51.161230 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:51.163523 12986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2: Bootstrap starting.
I20260928 21:59:51.164062 12986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:51.164302 12986 log.cc:824] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:51.165047 12986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2: No bootstrap required, opened a new log
I20260928 21:59:51.165949 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 12918
I20260928 21:59:51.166088 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 21:59:51.166332 2709 external_mini_cluster.cc:1428] Setting key ffaf76645459e28b9f080bfc90fcee1c
I20260928 21:59:51.166970 12986 raft_consensus.cc:348] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e324d82d38eb4208b864083fa03623b2" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45935 } }
I20260928 21:59:51.167189 12986 raft_consensus.cc:374] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:51.167227 12986 raft_consensus.cc:729] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e324d82d38eb4208b864083fa03623b2, State: Initialized, Role: FOLLOWER
I20260928 21:59:51.167364 12986 consensus_queue.cc:260] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e324d82d38eb4208b864083fa03623b2" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45935 } }
I20260928 21:59:51.167461 12986 raft_consensus.cc:388] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:59:51.167505 12986 raft_consensus.cc:482] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:59:51.167555 12986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:51.168453 12986 raft_consensus.cc:504] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e324d82d38eb4208b864083fa03623b2" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45935 } }
I20260928 21:59:51.168576 12986 leader_election.cc:302] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e324d82d38eb4208b864083fa03623b2; no voters:
I20260928 21:59:51.168756 12986 leader_election.cc:288] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:59:51.168784 12991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:51.168886 12991 raft_consensus.cc:686] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [term 1 LEADER]: Becoming Leader. State: Replica: e324d82d38eb4208b864083fa03623b2, State: Running, Role: LEADER
I20260928 21:59:51.168973 12991 consensus_queue.cc:237] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e324d82d38eb4208b864083fa03623b2" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45935 } }
I20260928 21:59:51.169035 12986 sys_catalog.cc:564] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:59:51.169286 12991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e324d82d38eb4208b864083fa03623b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e324d82d38eb4208b864083fa03623b2" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45935 } } }
I20260928 21:59:51.169355 12991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:51.169472 12991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e324d82d38eb4208b864083fa03623b2. Latest consensus state: current_term: 1 leader_uuid: "e324d82d38eb4208b864083fa03623b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e324d82d38eb4208b864083fa03623b2" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45935 } } }
I20260928 21:59:51.169543 12991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:59:51.170140 12995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:59:51.170617 12995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:59:51.172272 12995 catalog_manager.cc:1347] Generated new cluster ID: a6b19ec4c35a4145a02a46e62be1abac
I20260928 21:59:51.172331 12995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:51.178507 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.166969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.2.165.126:45935)
0928 21:59:51.167160 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.167170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.167344 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:59:51.167581 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.167589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.167653 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.167819 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:51.168448 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.169877 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.170918 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.171506 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.171613 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.174621 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:51.174625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:51.174626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:51.174645 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:51.176781 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.177194 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.177198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.177198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.177319 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.177541 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.177542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.177542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.177852 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:51.178007 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.178157 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.178308 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260928 21:59:51.195063 12995 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:59:51.195676 12995 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:59:51.201582 12995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2: Generated new TSK 0
I20260928 21:59:51.201805 12995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 21:59:51.233315 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45935
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:51.317536 13012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:51.317838 13012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:51.317870 13012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:51.319041 13012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:51.319079 13012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:51.319108 13012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 21:59:51.320557 13012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:51.320782 13012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:51.320977 13012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:51.326290 13017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:51.326296 13018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:51.326308 13020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:51.326480 13012 server_base.cc:1034] running on GCE node
I20260928 21:59:51.326792 13012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:51.327029 13012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:51.328176 13012 hybrid_clock.cc:648] HybridClock initialized: now 1790632791328163 us; error 35 us; skew 500 ppm
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:51.330878 13012 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 21:59:51.331357 13012 webserver.cc:460] Webserver started at http://127.2.165.65:45941/ using document root <none> and password file <none>
I20260928 21:59:51.331600 13012 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:51.331643 13012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:51.331732 13012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:51.332852 13012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "fbf8728ca0dc3b787d0fccb2a685b2d1"
server_key_iv: "e756b75d2f3f4edd02225860dbde7ad8"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.333158 13012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "fbf8728ca0dc3b787d0fccb2a685b2d1"
server_key_iv: "e756b75d2f3f4edd02225860dbde7ad8"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.334628 13012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:51.335505 13027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.335776 13012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:51.335891 13012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "fbf8728ca0dc3b787d0fccb2a685b2d1"
server_key_iv: "e756b75d2f3f4edd02225860dbde7ad8"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.335983 13012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:51.384107 13012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:51.386606 13012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:51.386785 13012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:51.387048 13012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:51.387410 13012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:51.387465 13012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.387508 13012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:51.387535 13012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.393814 13012 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:33535
I20260928 21:59:51.393839 13140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:33535 every 8 connection(s)
I20260928 21:59:51.394196 13012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 21:59:51.399062 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 13012
I20260928 21:59:51.399180 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 21:59:51.399647 2709 external_mini_cluster.cc:1428] Setting key d1d258a68af611525725e6988caf98fb
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:51.404600 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.394974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:56903 (local address 127.2.165.126:45935)
0928 21:59:51.395069 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.395071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.395248 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:59:51.395791 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.395792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.395810 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.395896 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:51.396993 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.397525 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.398539 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.398748 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.398836 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.401193 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:51.401195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:51.401195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:51.401204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:51.402889 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.403293 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.403294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.403295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.403490 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.403773 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.403774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.403775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.403941 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:51.404017 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.404335 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.404465 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:59:51.404632 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.394889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45935 (local address 127.2.165.65:56903)
0928 21:59:51.395104 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.395157 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.395680 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.395977 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.395981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.396611 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:51.396888 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.396895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.397629 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.397630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.398445 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.398447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.398853 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.398854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.399023 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.399587 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:51.399597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:51.401095 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:51.402974 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:51.402980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:51.402982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:51.403212 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:51.403587 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:51.403588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:51.403589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:51.403698 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:51.404018 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:51.404023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:51.404147 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.404288 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.404463 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260928 21:59:51.405521 13141 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45935
I20260928 21:59:51.405642 13141 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:51.405827 13141 heartbeater.cc:511] Master 127.2.165.126:45935 requested a full tablet report, sending...
I20260928 21:59:51.406359 12950 ts_manager.cc:194] Registered new tserver with Master: 30ccd41b8c2c4a8ab5ad50582c1cfe72 (127.2.165.65:33535)
I20260928 21:59:51.406977 12950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:56903
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 21:59:51.428646 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45935
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:51.510677 13146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:51.510905 13146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:51.510927 13146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:51.512030 13146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:51.512065 13146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:51.512092 13146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 21:59:51.513509 13146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:51.513700 13146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:51.513887 13146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:51.519150 13152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:51.519205 13146 server_base.cc:1034] running on GCE node
W20260928 21:59:51.519222 13154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:51.519202 13151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:51.519564 13146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:51.519845 13146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:51.521026 13146 hybrid_clock.cc:648] HybridClock initialized: now 1790632791521020 us; error 38 us; skew 500 ppm
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:51.523717 13146 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 21:59:51.524183 13146 webserver.cc:460] Webserver started at http://127.2.165.66:41149/ using document root <none> and password file <none>
I20260928 21:59:51.524447 13146 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:51.524495 13146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:51.524663 13146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:51.525846 13146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "799692d52b0947c79c5b27f8f3f32b21"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "eb29f3f1f6c61a817a267134a616818b"
server_key_iv: "040f2cf09e934ef1cb67c6dd6862267d"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.526222 13146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "799692d52b0947c79c5b27f8f3f32b21"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "eb29f3f1f6c61a817a267134a616818b"
server_key_iv: "040f2cf09e934ef1cb67c6dd6862267d"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.527673 13146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:59:51.528553 13161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.528851 13146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:59:51.528951 13146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "799692d52b0947c79c5b27f8f3f32b21"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "eb29f3f1f6c61a817a267134a616818b"
server_key_iv: "040f2cf09e934ef1cb67c6dd6862267d"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.529031 13146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:51.550940 13146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:51.553397 13146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:51.553555 13146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:51.553828 13146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:51.554160 13146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:51.554193 13146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.554217 13146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:51.554231 13146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.560029 13146 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:36263
I20260928 21:59:51.560097 13274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:36263 every 8 connection(s)
I20260928 21:59:51.560403 13146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 21:59:51.564064 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 13146
I20260928 21:59:51.564180 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 21:59:51.564376 2709 external_mini_cluster.cc:1428] Setting key c103d9dbdcec30ab500c5b1e8c3caba1
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:51.571681 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.561272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:53699 (local address 127.2.165.126:45935)
0928 21:59:51.561345 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.561348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.561593 (+ 245us) server_negotiation.cc:407] Connection header received
0928 21:59:51.562056 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.562058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.562069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.562160 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:51.563222 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.563779 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.565088 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.565472 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.565546 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.568403 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:51.568405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:51.568405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:51.568415 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:51.570185 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.570650 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.570652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.570652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.570715 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.570989 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.570990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.570990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.571117 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:51.571195 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.571494 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.571581 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:59:51.571907 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.561223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45935 (local address 127.2.165.66:53699)
0928 21:59:51.561446 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.561485 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.561964 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.562236 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.562239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.562853 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:51.563057 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.563063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.563912 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.563913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.564968 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.564971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.565573 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.565574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.565734 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.566400 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:51.566446 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:51.568277 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:51.570286 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:51.570295 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:51.570298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:51.570553 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:51.570793 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:51.570794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:51.570794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:51.570915 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:51.571185 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:51.571192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:51.571354 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.571476 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.571651 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260928 21:59:51.572598 13275 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45935
I20260928 21:59:51.572736 13275 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:51.572938 13275 heartbeater.cc:511] Master 127.2.165.126:45935 requested a full tablet report, sending...
I20260928 21:59:51.573393 12950 ts_manager.cc:194] Registered new tserver with Master: 799692d52b0947c79c5b27f8f3f32b21 (127.2.165.66:36263)
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 21:59:51.573915 12950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:53699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 21:59:51.591673 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45935
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:59:51.675045 13280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:59:51.675278 13280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:59:51.675303 13280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:59:51.676496 13280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:59:51.676535 13280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:59:51.676561 13280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 21:59:51.678040 13280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:59:51.678249 13280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:59:51.678442 13280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:59:51.683641 13285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:51.683636 13288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:59:51.683938 13286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:59:51.683938 13280 server_base.cc:1034] running on GCE node
I20260928 21:59:51.684149 13280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:59:51.684357 13280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:59:51.685527 13280 hybrid_clock.cc:648] HybridClock initialized: now 1790632791685515 us; error 43 us; skew 500 ppm
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:59:51.687985 13280 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 21:59:51.688465 13280 webserver.cc:460] Webserver started at http://127.2.165.67:41277/ using document root <none> and password file <none>
I20260928 21:59:51.688767 13280 fs_manager.cc:359] Metadata directory not provided
I20260928 21:59:51.688839 13280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:59:51.688994 13280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:59:51.690075 13280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "f863d5059d05487d913eb7f972330d7b"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "b4a11e72b2b931955fb0faf484c98d0f"
server_key_iv: "8f57f1ff99c6f75cd6ad5aa0a279ae7c"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.690448 13280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "f863d5059d05487d913eb7f972330d7b"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "b4a11e72b2b931955fb0faf484c98d0f"
server_key_iv: "8f57f1ff99c6f75cd6ad5aa0a279ae7c"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.691766 13280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:59:51.692667 13295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.692921 13280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:59:51.693030 13280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "f863d5059d05487d913eb7f972330d7b"
format_stamp: "Formatted at 2026-09-28 21:59:51 on dist-test-slave-3t36"
server_key: "b4a11e72b2b931955fb0faf484c98d0f"
server_key_iv: "8f57f1ff99c6f75cd6ad5aa0a279ae7c"
server_key_version: "encryptionkey@0"
I20260928 21:59:51.693120 13280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:59:51.721632 13280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:59:51.723701 13280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:59:51.723881 13280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:59:51.724129 13280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:59:51.724472 13280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:59:51.724529 13280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.724576 13280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:59:51.724608 13280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:59:51.730701 13280 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:37767
I20260928 21:59:51.730748 13408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:37767 every 8 connection(s)
I20260928 21:59:51.731087 13280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 21:59:51.737010 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 13280
I20260928 21:59:51.737125 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 21:59:51.737318 2709 external_mini_cluster.cc:1428] Setting key 9e8b345898931bbf759ad0deaee3a725
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[12905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 21:59:51.741104 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.731869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:58825 (local address 127.2.165.126:45935)
0928 21:59:51.731940 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.731942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.732184 (+ 242us) server_negotiation.cc:407] Connection header received
0928 21:59:51.732953 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.732954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.732966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.733015 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:51.734001 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.734521 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.735556 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.735771 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.735858 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.738262 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:51.738264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:51.738265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:51.738274 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:51.739603 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.740033 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.740035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.740035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.740100 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.740372 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.740373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.740374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.740499 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:51.740576 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.740934 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.741014 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:59:51.741304 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.731889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45935 (local address 127.2.165.67:58825)
0928 21:59:51.732048 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.732085 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.732820 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.733094 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.733098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.733690 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:59:51.733888 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.733893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.734632 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.734633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.735451 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.735454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.735867 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.735868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.736009 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.736540 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:59:51.736548 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:59:51.738153 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:59:51.739677 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:51.739684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:51.739686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:51.739945 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:51.740167 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:59:51.740169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:59:51.740169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:59:51.740293 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:59:51.740567 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:59:51.740573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:59:51.740777 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.740900 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.741063 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260928 21:59:51.741830 13409 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45935
I20260928 21:59:51.741932 13409 heartbeater.cc:464] Registering TS with master...
I20260928 21:59:51.742120 13409 heartbeater.cc:511] Master 127.2.165.126:45935 requested a full tablet report, sending...
I20260928 21:59:51.742552 12950 ts_manager.cc:194] Registered new tserver with Master: f863d5059d05487d913eb7f972330d7b (127.2.165.67:37767)
I20260928 21:59:51.742895 12950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:58825
I20260928 21:59:51.749292 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:59:51.757121 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.750687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.2.165.126:45935)
0928 21:59:51.750760 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.750762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.750860 (+ 98us) server_negotiation.cc:407] Connection header received
0928 21:59:51.750943 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.750945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.750955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.751033 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:59:51.751641 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.752130 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.752942 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.753140 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.753217 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.754473 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:59:51.754475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:59:51.754475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:59:51.754483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:59:51.755798 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.756212 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.756213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.756213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.756273 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:59:51.756477 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:59:51.756478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:59:51.756478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:59:51.756636 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:59:51.756776 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.756926 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.757012 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:59:51.758898 12950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:59:51.760569 12950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:59:51.767524 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.763141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:37767 (local address 127.0.0.1:35266)
0928 21:59:51.763490 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.763501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.763554 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.764144 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.764144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.764146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:51.764232 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.764233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.765560 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.765561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.766586 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.766587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.767271 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.767272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.767319 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.767321 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.767356 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.767409 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":28,"thread_start_us":141,"threads_started":1}
I20260928 21:59:51.767545 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.763589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.2.165.66:36263)
0928 21:59:51.763761 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.763764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.763772 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:59:51.763803 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.763805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.763816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.763900 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:51.764246 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.765025 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.766392 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.767121 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.767200 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.767261 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.767410 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.767465 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260928 21:59:51.767714 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.763108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:33535 (local address 127.0.0.1:41634)
0928 21:59:51.763353 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.763399 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.763468 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.763849 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.763851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.764038 (+ 187us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:51.764206 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.764208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.765592 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.765593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.766579 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.766582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.767485 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.767486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.767545 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.767549 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.767594 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.767633 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260928 21:59:51.767946 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.763306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.2.165.65:33535)
0928 21:59:51.763685 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.763691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.763702 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:51.763741 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.763743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.763760 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.763844 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:51.764296 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.765451 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.766674 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.767405 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.767471 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.767523 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.767821 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.767874 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":92,"threads_started":1}
I20260928 21:59:51.767984 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.763387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:36263 (local address 127.0.0.1:43652)
0928 21:59:51.763482 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.763492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.763532 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.763914 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.763915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.764063 (+ 148us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:51.764180 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.764183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.765119 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.765120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.766305 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.766307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.767211 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.767212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.767293 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.767309 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.767382 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.767413 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":7,"thread_start_us":62,"threads_started":1}
I20260928 21:59:51.768240 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.763264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.2.165.67:37767)
0928 21:59:51.763976 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.763980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.763989 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:59:51.764029 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.764031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.764046 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.764315 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:51.764378 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.765457 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.766670 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.767152 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.767877 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.767944 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.767968 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.768163 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":97,"threads_started":1}
I20260928 21:59:51.768746 13075 tablet_service.cc:1467] Processing CreateTablet for tablet fc789de5a6f64c888b176d1de6d296a4 (DEFAULT_TABLE table=test-table [id=2e767405c71749c4b9f345fbd85a3ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:51.768813 13209 tablet_service.cc:1467] Processing CreateTablet for tablet fc789de5a6f64c888b176d1de6d296a4 (DEFAULT_TABLE table=test-table [id=2e767405c71749c4b9f345fbd85a3ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:51.768934 13343 tablet_service.cc:1467] Processing CreateTablet for tablet fc789de5a6f64c888b176d1de6d296a4 (DEFAULT_TABLE table=test-table [id=2e767405c71749c4b9f345fbd85a3ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:59:51.769124 13075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc789de5a6f64c888b176d1de6d296a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:51.769124 13209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc789de5a6f64c888b176d1de6d296a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:51.769296 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc789de5a6f64c888b176d1de6d296a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:59:51.772092 13426 tablet_bootstrap.cc:492] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Bootstrap starting.
I20260928 21:59:51.772452 13428 tablet_bootstrap.cc:492] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Bootstrap starting.
I20260928 21:59:51.772827 13427 tablet_bootstrap.cc:492] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72: Bootstrap starting.
I20260928 21:59:51.772913 13426 tablet_bootstrap.cc:654] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:51.773101 13428 tablet_bootstrap.cc:654] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:51.773432 13427 tablet_bootstrap.cc:654] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72: Neither blocks nor log segments found. Creating new log.
I20260928 21:59:51.773515 13426 log.cc:824] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:51.773670 13428 log.cc:824] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:51.774083 13427 log.cc:824] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72: Log is configured to *not* fsync() on all Append() calls
I20260928 21:59:51.774405 13426 tablet_bootstrap.cc:492] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: No bootstrap required, opened a new log
I20260928 21:59:51.774483 13426 ts_tablet_manager.cc:1397] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:59:51.774523 13428 tablet_bootstrap.cc:492] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: No bootstrap required, opened a new log
I20260928 21:59:51.774592 13428 ts_tablet_manager.cc:1397] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:51.774806 13427 tablet_bootstrap.cc:492] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72: No bootstrap required, opened a new log
I20260928 21:59:51.774873 13427 ts_tablet_manager.cc:1397] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:59:51.776288 13428 raft_consensus.cc:348] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.776466 13428 raft_consensus.cc:374] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:51.776511 13428 raft_consensus.cc:729] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 799692d52b0947c79c5b27f8f3f32b21, State: Initialized, Role: FOLLOWER
I20260928 21:59:51.776497 13426 raft_consensus.cc:348] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.776692 13426 raft_consensus.cc:374] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:51.776652 13428 consensus_queue.cc:260] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.776742 13426 raft_consensus.cc:729] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f863d5059d05487d913eb7f972330d7b, State: Initialized, Role: FOLLOWER
I20260928 21:59:51.776860 13426 consensus_queue.cc:260] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.776975 13428 ts_tablet_manager.cc:1428] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:59:51.776889 13427 raft_consensus.cc:348] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.777020 13275 heartbeater.cc:503] Master 127.2.165.126:45935 was elected leader, sending a full tablet report...
I20260928 21:59:51.777035 13427 raft_consensus.cc:374] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:59:51.777071 13427 raft_consensus.cc:729] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ccd41b8c2c4a8ab5ad50582c1cfe72, State: Initialized, Role: FOLLOWER
I20260928 21:59:51.777136 13426 ts_tablet_manager.cc:1428] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:59:51.777180 13427 consensus_queue.cc:260] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.777278 13409 heartbeater.cc:503] Master 127.2.165.126:45935 was elected leader, sending a full tablet report...
I20260928 21:59:51.777454 13427 ts_tablet_manager.cc:1428] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:59:51.777599 13141 heartbeater.cc:503] Master 127.2.165.126:45935 was elected leader, sending a full tablet report...
W20260928 21:59:51.811059 13276 tablet.cc:2369] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:59:51.856400 13433 raft_consensus.cc:482] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:59:51.856526 13433 raft_consensus.cc:504] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.856982 13433 leader_election.cc:288] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ccd41b8c2c4a8ab5ad50582c1cfe72 (127.2.165.65:33535), 799692d52b0947c79c5b27f8f3f32b21 (127.2.165.66:36263)
I20260928 21:59:51.859985 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.857386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:35611 (local address 127.2.165.66:36263)
0928 21:59:51.857439 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.857441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.857452 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:51.857487 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.857488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.857497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.857593 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:51.857778 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.858263 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.859222 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.859602 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.859703 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.859713 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.859869 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.859909 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:59:51.860013 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.857183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:36263 (local address 127.2.165.67:35611)
0928 21:59:51.857257 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.857269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.857317 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.857600 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.857602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.857603 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:51.857700 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.857702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.858358 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.858359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.859112 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.859114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.859701 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.859702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.859788 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.859792 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.859863 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.859931 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":20}
I20260928 21:59:51.860026 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.857302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:33535 (local address 127.2.165.67:50743)
0928 21:59:51.857480 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.857488 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:59:51.857551 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:59:51.857700 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:59:51.857700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:59:51.857702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:59:51.857768 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.857770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.858410 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.858411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.859179 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:59:51.859180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.859716 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.859717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:59:51.859803 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.859806 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:59:51.859881 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:59:51.859937 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":3,"thread_start_us":125,"threads_started":1}
I20260928 21:59:51.860003 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.857356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:50743 (local address 127.2.165.65:33535)
0928 21:59:51.857407 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.857409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.857543 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:59:51.857616 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.857619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.857632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.857697 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:59:51.857846 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.858326 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.859271 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.859632 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.859717 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.859725 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.859891 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.859934 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:59:51.860348 13229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc789de5a6f64c888b176d1de6d296a4" candidate_uuid: "f863d5059d05487d913eb7f972330d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "799692d52b0947c79c5b27f8f3f32b21" is_pre_election: true
I20260928 21:59:51.860474 13095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc789de5a6f64c888b176d1de6d296a4" candidate_uuid: "f863d5059d05487d913eb7f972330d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" is_pre_election: true
I20260928 21:59:51.860524 13229 raft_consensus.cc:2454] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f863d5059d05487d913eb7f972330d7b in term 0.
I20260928 21:59:51.860602 13095 raft_consensus.cc:2454] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f863d5059d05487d913eb7f972330d7b in term 0.
I20260928 21:59:51.860781 13299 leader_election.cc:302] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 799692d52b0947c79c5b27f8f3f32b21, f863d5059d05487d913eb7f972330d7b; no voters:
I20260928 21:59:51.861048 13433 raft_consensus.cc:2781] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:59:51.861112 13433 raft_consensus.cc:482] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:59:51.861140 13433 raft_consensus.cc:3037] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:51.861861 13433 raft_consensus.cc:504] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.861997 13433 leader_election.cc:288] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [CANDIDATE]: Term 1 election: Requested vote from peers 30ccd41b8c2c4a8ab5ad50582c1cfe72 (127.2.165.65:33535), 799692d52b0947c79c5b27f8f3f32b21 (127.2.165.66:36263)
I20260928 21:59:51.862203 13229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc789de5a6f64c888b176d1de6d296a4" candidate_uuid: "f863d5059d05487d913eb7f972330d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "799692d52b0947c79c5b27f8f3f32b21"
I20260928 21:59:51.862210 13095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc789de5a6f64c888b176d1de6d296a4" candidate_uuid: "f863d5059d05487d913eb7f972330d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72"
I20260928 21:59:51.862294 13229 raft_consensus.cc:3037] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:51.862294 13095 raft_consensus.cc:3037] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:59:51.863205 13229 raft_consensus.cc:2454] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f863d5059d05487d913eb7f972330d7b in term 1.
I20260928 21:59:51.863214 13095 raft_consensus.cc:2454] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f863d5059d05487d913eb7f972330d7b in term 1.
I20260928 21:59:51.863370 13299 leader_election.cc:302] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 799692d52b0947c79c5b27f8f3f32b21, f863d5059d05487d913eb7f972330d7b; no voters:
I20260928 21:59:51.863461 13433 raft_consensus.cc:2781] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:59:51.863610 13433 raft_consensus.cc:686] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 1 LEADER]: Becoming Leader. State: Replica: f863d5059d05487d913eb7f972330d7b, State: Running, Role: LEADER
I20260928 21:59:51.863711 13433 consensus_queue.cc:237] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } }
I20260928 21:59:51.864502 12950 catalog_manager.cc:5495] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b reported cstate change: term changed from 0 to 1, leader changed from <none> to f863d5059d05487d913eb7f972330d7b (127.2.165.67). New cstate: current_term: 1 leader_uuid: "f863d5059d05487d913eb7f972330d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f863d5059d05487d913eb7f972330d7b" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 } health_report { overall_health: UNKNOWN } } }
I20260928 21:59:51.888460 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.885911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.2.165.67:37767)
0928 21:59:51.885973 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:59:51.885975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:59:51.885986 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:59:51.886054 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:59:51.886055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:59:51.886078 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:59:51.886169 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:59:51.886341 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.886831 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.887588 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:59:51.887830 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:59:51.887913 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:59:51.888073 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:59:51.888189 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:59:51.888307 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:59:51.888327 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 21:59:51.888349 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:59:51.890376 13229 raft_consensus.cc:1261] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 1 FOLLOWER]: Refusing update from remote peer f863d5059d05487d913eb7f972330d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:51.890404 13095 raft_consensus.cc:1261] T fc789de5a6f64c888b176d1de6d296a4 P 30ccd41b8c2c4a8ab5ad50582c1cfe72 [term 1 FOLLOWER]: Refusing update from remote peer f863d5059d05487d913eb7f972330d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:59:51.890738 13433 consensus_queue.cc:1035] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "799692d52b0947c79c5b27f8f3f32b21" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:51.890901 13436 consensus_queue.cc:1035] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ccd41b8c2c4a8ab5ad50582c1cfe72" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:59:51.893275 13439 mvcc.cc:205] Tried to move back new op lower bound from 7334431915580116992 to 7334431915474456576. Current Snapshot: MvccSnapshot[applied={T|T < 7334431915580116992}]
I20260928 21:59:51.893573 13440 mvcc.cc:205] Tried to move back new op lower bound from 7334431915580116992 to 7334431915474456576. Current Snapshot: MvccSnapshot[applied={T|T < 7334431915580116992}]
I20260928 21:59:51.894029 13441 mvcc.cc:205] Tried to move back new op lower bound from 7334431915580116992 to 7334431915474456576. Current Snapshot: MvccSnapshot[applied={T|T < 7334431915580116992}]
I20260928 21:59:51.895923 12950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:51.896030 12950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:59:51.896873 12950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e324d82d38eb4208b864083fa03623b2: Sending DeleteTablet for 3 replicas of tablet fc789de5a6f64c888b176d1de6d296a4
I20260928 21:59:51.897220 13075 tablet_service.cc:1514] Processing DeleteTablet for tablet fc789de5a6f64c888b176d1de6d296a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:51 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:41634
I20260928 21:59:51.897395 13209 tablet_service.cc:1514] Processing DeleteTablet for tablet fc789de5a6f64c888b176d1de6d296a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:51 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:43652
I20260928 21:59:51.897413 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet fc789de5a6f64c888b176d1de6d296a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:59:51 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35266
I20260928 21:59:51.897630 13449 tablet_replica.cc:330] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: stopping tablet replica
I20260928 21:59:51.897639 13450 tablet_replica.cc:330] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: stopping tablet replica
I20260928 21:59:51.897735 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 13012
I20260928 21:59:51.897768 13450 raft_consensus.cc:2229] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:59:51.897768 13449 raft_consensus.cc:2229] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:59:51.897879 13450 raft_consensus.cc:2258] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:51.898007 13449 raft_consensus.cc:2258] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:59:51.898690 13450 ts_tablet_manager.cc:1905] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:51.898690 13449 ts_tablet_manager.cc:1905] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:59:51.900651 13450 ts_tablet_manager.cc:1918] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:51.900651 13449 ts_tablet_manager.cc:1918] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:59:51.900765 13450 log.cc:1196] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/fc789de5a6f64c888b176d1de6d296a4
I20260928 21:59:51.900884 13449 log.cc:1196] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/fc789de5a6f64c888b176d1de6d296a4
I20260928 21:59:51.901067 13450 ts_tablet_manager.cc:1939] T fc789de5a6f64c888b176d1de6d296a4 P 799692d52b0947c79c5b27f8f3f32b21: Deleting consensus metadata
I20260928 21:59:51.901499 12937 catalog_manager.cc:4841] TS 799692d52b0947c79c5b27f8f3f32b21 (127.2.165.66:36263): tablet fc789de5a6f64c888b176d1de6d296a4 (table test-table [id=2e767405c71749c4b9f345fbd85a3ed1]) successfully deleted
I20260928 21:59:51.902321 13449 ts_tablet_manager.cc:1939] T fc789de5a6f64c888b176d1de6d296a4 P f863d5059d05487d913eb7f972330d7b: Deleting consensus metadata
I20260928 21:59:51.902741 12937 catalog_manager.cc:4841] TS f863d5059d05487d913eb7f972330d7b (127.2.165.67:37767): tablet fc789de5a6f64c888b176d1de6d296a4 (table test-table [id=2e767405c71749c4b9f345fbd85a3ed1]) successfully deleted
W20260928 21:59:51.906205 12936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:33535 (error 108)
I20260928 21:59:51.906513 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 13146
I20260928 21:59:51.907141 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:59:51.906671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:33535 (local address 127.0.0.1:41640)
0928 21:59:51.907067 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 21:59:51.907104 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377}
W20260928 21:59:51.907508 12936 catalog_manager.cc:4584] TS 30ccd41b8c2c4a8ab5ad50582c1cfe72 (127.2.165.65:33535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc789de5a6f64c888b176d1de6d296a4: Network error: Client connection negotiation failed: client connection to 127.2.165.65:33535: connect: Connection refused (error 111)
I20260928 21:59:51.912271 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 13280
I20260928 21:59:51.917497 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 12918
2026-09-28T21:59:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2050 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[13454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[13454](info): set up 2 sockets
Sep 28 21:59:51 dist-test-slave-3t36 krb5kdc[13454](info): commencing operation
krb5kdc: starting...
W20260928 21:59:52.965761 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 21:59:53 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:59:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/rangeradmin_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/rangeradmin_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/rangerlookup_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/rangerlookup_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
I20260928 21:59:53.144341 2709 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjhUuNU/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/postgres -l logfile start
2026-09-28 21:59:55.970 UTC [13484] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:59:55.970 UTC [13484] LOG: listening on IPv4 address "127.2.165.84", port 56691
2026-09-28 21:59:55.974 UTC [13484] LOG: listening on Unix socket "/tmp/.s.PGSQL.56691"
2026-09-28 21:59:55.997 UTC [13487] LOG: database system was shut down at 2026-09-28 21:59:54 UTC
2026-09-28 21:59:56.001 UTC [13484] LOG: database system is ready to accept connections
I20260928 21:59:56.949503 2709 mini_postgres.cc:96] Postgres bound to 56691
2026-09-28 21:59:56.955 UTC [13495] FATAL: database "slave" does not exist
127.2.165.84:56691 - accepting connections
I20260928 21:59:56.956460 2709 mini_ranger.cc:161] Starting Ranger...
I20260928 21:59:56.973008 2709 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:59:57.557367 2709 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:59:57.557533 2709 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-admin
2026-09-28 21:59:57,839 [I] DB FLAVOR :POSTGRES
2026-09-28 21:59:57,840 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:59:57,840 [I] Checking connection..
2026-09-28 21:59:57,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:59:58,103 [I] Checking connection passed.
2026-09-28 21:59:58,104 [I] --------- Verifying version history table ---------
2026-09-28 21:59:58,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:59:58,353 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:59:58,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:59:58,611 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:59:58,611 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:59:58,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:59:58.868 UTC [13559] WARNING: there is no transaction in progress
2026-09-28 21:59:58,874 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:59:58,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:59:59,126 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:59:59,126 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:59:59,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:59:59,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:59:59,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3t36.c.gcp-upstream.internal','N') ;"
2026-09-28 21:59:59,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:00:00,157 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:00:00,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:00:00,418 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:00:00,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:00:00,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:00:00,925 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:00:00,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:00:02.056 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.076 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.097 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.118 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.137 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.160 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.225 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.233 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.252 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.265 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.280 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.299 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.311 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.319 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.329 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:02.339 UTC [13667] WARNING: there is no transaction in progress
2026-09-28 22:00:03,162 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:00:03,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:00:03,409 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:00:03,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:00:03,668 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:00:03,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:00:03,931 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:00:03,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:00:04,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:00:04,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3t36.c.gcp-upstream.internal';"
2026-09-28 22:00:04,739 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:00:04,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:00:05,005 [I] DB_PATCHES have already been applied
I20260928 22:00:05.010998 2709 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjhUuNU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:00:05.011072 2709 mini_ranger.cc:193] Using host: 127.2.165.84
I20260928 22:00:05.014485 2709 mini_ranger.cc:227] Ranger admin URL: http://127.2.165.84:52269
Sep 28, 2026 10:00:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:00:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:00:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:00:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:00:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:00:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:00:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:00:06.179224 2709 mini_ranger.cc:160] Time spent starting Ranger: real 9.223s user 0.000s sys 0.009s
Sep 28, 2026 10:00:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:00:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 10:00:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:00:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:00:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:00:29.314592 2709 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/rangerkms_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/rangerkms_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.165.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.165.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:00:29 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632829, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:00:29.402330 2709 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:00:29.412111 2709 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:00:30.382956 2709 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:00:30.383047 2709 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms
I20260928 22:00:30.383069 2709 mini_ranger_kms.cc:227] Using postgres at 127.2.165.84:56691
2026-09-28 22:00:30,573 [I] DB FLAVOR :POSTGRES
2026-09-28 22:00:30,573 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:00:30,573 [I] Checking connection
2026-09-28 22:00:30,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:00:30,818 [I] connection success
2026-09-28 22:00:30,818 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:00:30,818 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:00:30,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:00:31,089 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:00:31,090 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:00:31,090 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:00:31,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:56691/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:00:31,355 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:00:31.912973 2709 mini_ranger_kms.cc:310] Created kms service
I20260928 22:00:32.111367 2709 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:00:32.161974 2709 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:00:32 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632829, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:00:32.586110 2709 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:00:32.586228 2709 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:00:32.586249 2709 mini_ranger_kms.cc:240] Using host: 127.2.165.84
I20260928 22:00:32.589756 2709 mini_ranger_kms.cc:276] Ranger KMS PID: 13848
I20260928 22:00:32.589888 2709 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.165.84:52855
Sep 28, 2026 10:00:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:00:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:00:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:00:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:00:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:00:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:00:34.936565 2709 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.534s user 0.006s sys 0.165s
I20260928 22:00:34.936717 2709 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:00:34.936808 2709 mini_ranger_kms.cc:401] 127.2.165.84:52855/kms/v1/keys
Sep 28 22:00:35 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.165.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:00:35 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.165.84@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
Sep 28 22:00:35 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.165.84@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
Sep 28 22:00:38 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:00:38.252245 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:39689
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:39689
--ranger_config_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:52855/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:00:38.339479 13898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:00:38.339705 13898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:00:38.339728 13898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:00:38.341141 13898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:00:38.341197 13898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:00:38.341212 13898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:00:38.341225 13898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:00:38.341238 13898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:00:38.342972 13898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.165.84:52855/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:39689
--ranger_config_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:00:38.343253 13898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:00:38.343544 13898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:00:38.349454 13903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:00:38.349587 13898 server_base.cc:1034] running on GCE node
W20260928 22:00:38.349475 13906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:00:38.349454 13904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:00:38.349915 13898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:00:38.350256 13898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:00:38.351476 13898 hybrid_clock.cc:648] HybridClock initialized: now 1790632838351446 us; error 47 us; skew 500 ppm
Sep 28 22:00:38 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:00:38.354228 13898 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:00:38.354803 13898 webserver.cc:460] Webserver started at http://127.2.165.126:32989/ using document root <none> and password file <none>
I20260928 22:00:38.355068 13898 fs_manager.cc:359] Metadata directory not provided
I20260928 22:00:38.355111 13898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:00:38.355255 13898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:00:38 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:00:38.420094 13898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c"
format_stamp: "Formatted at 2026-09-28 22:00:38 on dist-test-slave-3t36"
server_key: "dbfc08b0718161816f4d5d4465abb7e9"
server_key_iv: "977227703c8087df9499a93b8b21cdd6"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:38.420571 13898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c"
format_stamp: "Formatted at 2026-09-28 22:00:38 on dist-test-slave-3t36"
server_key: "dbfc08b0718161816f4d5d4465abb7e9"
server_key_iv: "977227703c8087df9499a93b8b21cdd6"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:38.422107 13898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:00:38.438426 13913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:38.438714 13898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:00:38.438835 13898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c"
format_stamp: "Formatted at 2026-09-28 22:00:38 on dist-test-slave-3t36"
server_key: "dbfc08b0718161816f4d5d4465abb7e9"
server_key_iv: "977227703c8087df9499a93b8b21cdd6"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:38.438978 13898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:00:38.472311 13898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:00:38.474658 13898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:00:38.474855 13898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:00:38.480075 13898 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:39689
I20260928 22:00:38.480106 13965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:39689 every 8 connection(s)
I20260928 22:00:38.480535 13898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:00:38.481292 13966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:00:38.484026 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c: Bootstrap starting.
I20260928 22:00:38.484738 13966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c: Neither blocks nor log segments found. Creating new log.
I20260928 22:00:38.485080 13966 log.cc:824] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:00:38.485931 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c: No bootstrap required, opened a new log
I20260928 22:00:38.488161 13966 raft_consensus.cc:348] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39689 } }
I20260928 22:00:38.488330 13966 raft_consensus.cc:374] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:00:38.488353 13966 raft_consensus.cc:729] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f230aef4ec3346fcbe7ddbe50fa69e1c, State: Initialized, Role: FOLLOWER
I20260928 22:00:38.488515 13966 consensus_queue.cc:260] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39689 } }
I20260928 22:00:38.488600 13966 raft_consensus.cc:388] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:00:38.488656 13966 raft_consensus.cc:482] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:00:38.488731 13966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:00:38.489406 13966 raft_consensus.cc:504] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39689 } }
I20260928 22:00:38.489531 13966 leader_election.cc:302] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f230aef4ec3346fcbe7ddbe50fa69e1c; no voters:
I20260928 22:00:38.489545 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 13898
I20260928 22:00:38.489630 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:00:38.489724 13966 leader_election.cc:288] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:00:38.489787 13969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:00:38.489847 2709 external_mini_cluster.cc:1428] Setting key f1d6229a5bab4bab4567776e4f819dc3
I20260928 22:00:38.490033 13969 raft_consensus.cc:686] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [term 1 LEADER]: Becoming Leader. State: Replica: f230aef4ec3346fcbe7ddbe50fa69e1c, State: Running, Role: LEADER
I20260928 22:00:38.490206 13969 consensus_queue.cc:237] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39689 } }
I20260928 22:00:38.490031 13966 sys_catalog.cc:564] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:00:38.491964 13966 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:00:38.493240 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f230aef4ec3346fcbe7ddbe50fa69e1c. Latest consensus state: current_term: 1 leader_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39689 } } }
I20260928 22:00:38.495141 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [sys.catalog]: This master's current role is: LEADER
I20260928 22:00:38.495098 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f230aef4ec3346fcbe7ddbe50fa69e1c" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 39689 } } }
I20260928 22:00:38.495285 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c [sys.catalog]: This master's current role is: LEADER
I20260928 22:00:38.495946 13988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:00:38.496925 13988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:00:38.499086 13988 catalog_manager.cc:1347] Generated new cluster ID: 9e1f0595861d4e6585778c6902b85184
I20260928 22:00:38.499142 13988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:00:38 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:00:38.508230 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:38.490360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.2.165.126:39689)
0928 22:00:38.490598 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:38.490608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:38.490812 (+ 204us) server_negotiation.cc:407] Connection header received
0928 22:00:38.490980 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:38.490989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:38.491055 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:38.491313 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:00:38.492595 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:38.496568 (+ 3973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:38.498055 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:38.498874 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:38.499047 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:38.502339 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:00:38.502346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:00:38.502347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:00:38.502365 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:00:38.505375 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:38.506007 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:38.506013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:38.506014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:38.506165 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:38.506400 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:38.506401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:38.506402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:38.506790 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:00:38.507000 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:38.507748 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 22:00:38.507916 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260928 22:00:38.522946 13988 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:00:38.523761 13988 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:00:38.531559 13988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c: Generated new TSK 0
I20260928 22:00:38.531795 13988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 22:00:39.241307 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:52855/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39689
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:00:39.351727 14011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:00:39.351991 14011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:00:39.352032 14011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:00:39.353431 14011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:00:39.353508 14011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:00:39.353551 14011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:00:39.355872 14011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.165.84:52855/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:00:39.356199 14011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:00:39.356575 14011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:00:39.363565 14019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:00:39.363744 14011 server_base.cc:1034] running on GCE node
W20260928 22:00:39.363854 14016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:00:39.363752 14017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:00:39.364152 14011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:00:39.364460 14011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:00:39.365650 14011 hybrid_clock.cc:648] HybridClock initialized: now 1790632839365624 us; error 37 us; skew 500 ppm
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:00:39.369175 14011 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:00:39.369793 14011 webserver.cc:460] Webserver started at http://127.2.165.65:42783/ using document root <none> and password file <none>
I20260928 22:00:39.370110 14011 fs_manager.cc:359] Metadata directory not provided
I20260928 22:00:39.370184 14011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:00:39.370334 14011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:00:39.385375 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "b2880e93caa14dafaf4ede04760e9795"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "367ef2fa59dabd9a01c84d2d2f565ca3"
server_key_iv: "5dd0637a316134402e839d7f903cea26"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.385929 14011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "b2880e93caa14dafaf4ede04760e9795"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "367ef2fa59dabd9a01c84d2d2f565ca3"
server_key_iv: "5dd0637a316134402e839d7f903cea26"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.389079 14011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 22:00:39.403313 14026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:39.403518 14011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:00:39.403635 14011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "b2880e93caa14dafaf4ede04760e9795"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "367ef2fa59dabd9a01c84d2d2f565ca3"
server_key_iv: "5dd0637a316134402e839d7f903cea26"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.403757 14011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:00:39.419263 14011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:00:39.421990 14011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:00:39.422224 14011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:00:39.422540 14011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:00:39.422923 14011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:00:39.422986 14011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:39.423031 14011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:00:39.423059 14011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:39.430579 14011 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:43153
I20260928 22:00:39.430675 14141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:43153 every 8 connection(s)
I20260928 22:00:39.431154 14011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:00:39.438840 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 14011
I20260928 22:00:39.438968 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:00:39.439203 2709 external_mini_cluster.cc:1428] Setting key 1c54d8d073f097b02be26707057c7689
I20260928 22:00:39.443341 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:39.432070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39689 (local address 127.2.165.65:43513)
0928 22:00:39.432249 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:39.432300 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:39.432945 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:39.433227 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:39.433231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:39.433423 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:00:39.433638 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:39.433642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.434498 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.434499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:39.435705 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:39.435708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.436204 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.436204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:39.436307 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:39.436927 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:00:39.436937 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:00:39.439024 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:00:39.441407 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:00:39.441418 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:00:39.441421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:00:39.441785 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:00:39.442037 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:00:39.442038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:00:39.442038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:00:39.442213 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:00:39.442566 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:00:39.442573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:00:39.442761 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 22:00:39.442887 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 22:00:39.443072 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 22:00:39.443946 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:39.432098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:43513 (local address 127.2.165.126:39689)
0928 22:00:39.432257 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:39.432259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:39.432399 (+ 140us) server_negotiation.cc:407] Connection header received
0928 22:00:39.433058 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:39.433060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:39.433075 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:39.433149 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:00:39.433749 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.434383 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.435781 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.436105 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.436193 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:39.439150 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:00:39.439152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:00:39.439152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:00:39.439162 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:00:39.441275 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:39.441884 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:39.441886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:39.441886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:39.441959 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:39.442297 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:39.442298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:39.442299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:39.442469 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:00:39.442537 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:39.443745 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0928 22:00:39.443837 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260928 22:00:39.444818 14142 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39689
I20260928 22:00:39.444938 14142 heartbeater.cc:464] Registering TS with master...
I20260928 22:00:39.445163 14142 heartbeater.cc:511] Master 127.2.165.126:39689 requested a full tablet report, sending...
I20260928 22:00:39.445803 13930 ts_manager.cc:194] Registered new tserver with Master: b2880e93caa14dafaf4ede04760e9795 (127.2.165.65:43153)
I20260928 22:00:39.446714 13930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:43513
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 22:00:39.473104 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:52855/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39689
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:00:39.584022 14148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:00:39.584272 14148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:00:39.584323 14148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:00:39.585624 14148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:00:39.585700 14148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:00:39.585744 14148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:00:39.587369 14148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.165.84:52855/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:00:39.587659 14148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:00:39.587983 14148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:00:39.593761 14158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:00:39.594092 14156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:00:39.594363 14148 server_base.cc:1034] running on GCE node
W20260928 22:00:39.594480 14155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:00:39.594718 14148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:00:39.595026 14148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:00:39.598321 14148 hybrid_clock.cc:648] HybridClock initialized: now 1790632839598295 us; error 42 us; skew 500 ppm
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:00:39.601388 14148 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:00:39.602016 14148 webserver.cc:460] Webserver started at http://127.2.165.66:41125/ using document root <none> and password file <none>
I20260928 22:00:39.602360 14148 fs_manager.cc:359] Metadata directory not provided
I20260928 22:00:39.602464 14148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:00:39.602620 14148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:00:39.624748 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "44b669582825447e9f746b0193f94253"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "f8369a910acac997432c2c3d9cacf974"
server_key_iv: "7e695476492c366bb43ce6df3c1225fa"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.625253 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "44b669582825447e9f746b0193f94253"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "f8369a910acac997432c2c3d9cacf974"
server_key_iv: "7e695476492c366bb43ce6df3c1225fa"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.628634 14148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 22:00:39.655534 14165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:39.655755 14148 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.000s
I20260928 22:00:39.655849 14148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "44b669582825447e9f746b0193f94253"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "f8369a910acac997432c2c3d9cacf974"
server_key_iv: "7e695476492c366bb43ce6df3c1225fa"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.655965 14148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:00:39.691063 14148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:00:39.693730 14148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:00:39.693902 14148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:00:39.694109 14148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:00:39.694505 14148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:00:39.694561 14148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:39.694598 14148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:00:39.694623 14148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:39.701603 14148 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:46189
I20260928 22:00:39.702069 14148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:00:39.705519 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 14148
I20260928 22:00:39.705643 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:00:39.705834 2709 external_mini_cluster.cc:1428] Setting key d21cb0bb20e0e3bd69060617b686d35e
I20260928 22:00:39.708954 14278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:46189 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:00:39.732796 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:39.709511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:43389 (local address 127.2.165.126:39689)
0928 22:00:39.709566 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:39.709569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:39.718328 (+ 8759us) server_negotiation.cc:407] Connection header received
0928 22:00:39.719286 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:39.719288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:39.719308 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:39.719353 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:00:39.720367 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.721209 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.725823 (+ 4614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.726152 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.726202 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:39.729177 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:00:39.729179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:00:39.729180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:00:39.729189 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:00:39.730980 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:39.731549 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:39.731551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:39.731551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:39.731616 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:39.731894 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:39.731895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:39.731895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:39.732041 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:00:39.732094 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:39.732520 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 22:00:39.732619 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:00:39.733816 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:39.709176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39689 (local address 127.2.165.66:43389)
0928 22:00:39.718106 (+ 8930us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:39.718189 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:39.719140 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:39.719571 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:39.719576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:39.719901 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:00:39.720206 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:39.720214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.724212 (+ 3998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.724214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:39.725688 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:39.725693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:39.726389 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:39.726390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:39.726574 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:39.727239 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:00:39.727250 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:00:39.729042 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:00:39.731092 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:00:39.731101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:00:39.731105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:00:39.731450 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:00:39.731680 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:00:39.731681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:00:39.731681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:00:39.731814 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:00:39.732129 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:00:39.732136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:00:39.732344 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 22:00:39.733348 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0928 22:00:39.733563 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8858,"thread_start_us":112,"threads_started":1}
I20260928 22:00:39.734694 14279 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39689
I20260928 22:00:39.734838 14279 heartbeater.cc:464] Registering TS with master...
I20260928 22:00:39.735081 14279 heartbeater.cc:511] Master 127.2.165.126:39689 requested a full tablet report, sending...
I20260928 22:00:39.735459 13930 ts_manager.cc:194] Registered new tserver with Master: 44b669582825447e9f746b0193f94253 (127.2.165.66:46189)
I20260928 22:00:39.735981 13930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:43389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 22:00:39.756059 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:52855/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39689
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:00:39.886767 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:00:39.887290 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:00:39.887399 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:00:39.889596 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:00:39.893361 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:00:39.893533 14285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:00:39.896418 14285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.165.84:52855/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:39689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:00:39.896986 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:00:39.897337 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:00:39.905462 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:00:39.905686 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:00:39.906145 14290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:00:39.907492 14285 server_base.cc:1034] running on GCE node
I20260928 22:00:39.907847 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:00:39.908154 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:00:39.909313 14285 hybrid_clock.cc:648] HybridClock initialized: now 1790632839909277 us; error 54 us; skew 500 ppm
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:00:39.912927 14285 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:00:39.913472 14285 webserver.cc:460] Webserver started at http://127.2.165.67:39739/ using document root <none> and password file <none>
I20260928 22:00:39.913748 14285 fs_manager.cc:359] Metadata directory not provided
I20260928 22:00:39.913797 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:00:39.913939 14285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:00:39 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:00:39.930003 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "565314e9463746cab1f1c131d473c59e"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "729f9ae3d587158c87009724e069f2bc"
server_key_iv: "8f30b305b33e24b8f2bf30ae4b2c6118"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.930508 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "565314e9463746cab1f1c131d473c59e"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "729f9ae3d587158c87009724e069f2bc"
server_key_iv: "8f30b305b33e24b8f2bf30ae4b2c6118"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.937757 14285 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.001s
I20260928 22:00:39.953670 14300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:39.953888 14285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 22:00:39.953994 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "565314e9463746cab1f1c131d473c59e"
format_stamp: "Formatted at 2026-09-28 22:00:39 on dist-test-slave-3t36"
server_key: "729f9ae3d587158c87009724e069f2bc"
server_key_iv: "8f30b305b33e24b8f2bf30ae4b2c6118"
server_key_version: "kuduclusterkey@0"
I20260928 22:00:39.954108 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:00:40.023976 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:00:40.026626 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:00:40.026796 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:00:40.027103 14285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:00:40.027530 14285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:00:40.027568 14285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:40.027609 14285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:00:40.027639 14285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:00:40.034441 14285 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:35867
I20260928 22:00:40.034911 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:00:40.037437 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 14285
I20260928 22:00:40.037555 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:00:40.037750 2709 external_mini_cluster.cc:1428] Setting key 58b5b0c9ffad3fa6ad2abd0eca43d896
I20260928 22:00:40.045802 14413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:35867 every 8 connection(s)
Sep 28 22:00:40 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:00:40.059866 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.046569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:43269 (local address 127.2.165.126:39689)
0928 22:00:40.046620 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.046622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.046936 (+ 314us) server_negotiation.cc:407] Connection header received
0928 22:00:40.047677 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.047679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.047692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.047725 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:00:40.048653 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.049404 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.051903 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.052206 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.052260 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.056105 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:00:40.056108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:00:40.056108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:00:40.056118 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:00:40.058024 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:40.058637 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:40.058639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:40.058639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:40.058703 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:40.059011 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:40.059013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:40.059013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:40.059161 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:00:40.059207 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.059641 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.059746 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:00:40.060823 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.046563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:39689 (local address 127.2.165.67:43269)
0928 22:00:40.046778 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.046841 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:40.047539 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:40.047933 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:40.047938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:40.048216 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:00:40.048530 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.048538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.049534 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.049535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.051765 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.051769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.052365 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.052366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.052539 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.053266 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:00:40.053278 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:00:40.055404 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:00:40.058147 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:00:40.058157 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:00:40.058160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:00:40.058532 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:00:40.058779 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:00:40.058779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:00:40.058780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:00:40.058927 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:00:40.059258 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:00:40.059265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:00:40.059471 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 22:00:40.060267 (+ 796us) client_negotiation.cc:240] Negotiation successful
0928 22:00:40.060482 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260928 22:00:40.061594 14414 heartbeater.cc:347] Connected to a master server at 127.2.165.126:39689
I20260928 22:00:40.061697 14414 heartbeater.cc:464] Registering TS with master...
I20260928 22:00:40.061928 14414 heartbeater.cc:511] Master 127.2.165.126:39689 requested a full tablet report, sending...
I20260928 22:00:40.062433 13930 ts_manager.cc:194] Registered new tserver with Master: 565314e9463746cab1f1c131d473c59e (127.2.165.67:35867)
I20260928 22:00:40.062932 13930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:43269
I20260928 22:00:40.070871 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:00:40.083071 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.072803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.2.165.126:39689)
0928 22:00:40.072849 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.072852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.075099 (+ 2247us) server_negotiation.cc:407] Connection header received
0928 22:00:40.075220 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.075221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.075233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.075270 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:00:40.076159 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.076967 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.078047 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.078331 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.078375 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.080019 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:00:40.080020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:00:40.080021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:00:40.080030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:00:40.081839 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:40.082236 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:40.082237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:40.082238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:40.082291 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:00:40.082475 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:00:40.082476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:00:40.082476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:00:40.082616 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:00:40.082689 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.082890 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.082967 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:00:40.085381 13930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:00:40.087913 13930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:00:40.102156 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.091295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:43153 (local address 127.0.0.1:37316)
0928 22:00:40.092471 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.092531 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:40.092590 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:40.093037 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:40.093040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:40.093042 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:00:40.093123 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.093125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.098311 (+ 5186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.098312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.099749 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.099751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.101832 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.101833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.101924 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.101937 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:00:40.102005 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:00:40.102060 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":74,"threads_started":1}
I20260928 22:00:40.103086 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.091531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:46189 (local address 127.0.0.1:35544)
0928 22:00:40.094330 (+ 2799us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.094341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:40.094386 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:40.094660 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:40.094661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:40.094663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:00:40.094709 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.094710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.095961 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.095962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.097805 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.097807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.102856 (+ 5049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.102857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.102925 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.102929 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:00:40.102974 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:00:40.103014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2781,"thread_start_us":67,"threads_started":1}
I20260928 22:00:40.103674 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.092045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.2.165.65:43153)
0928 22:00:40.092823 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.092826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.092837 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:00:40.092871 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.092873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.092890 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.092958 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:00:40.093199 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.098224 (+ 5025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.099847 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.100648 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.100742 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.100808 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.103475 (+ 2667us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.103595 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":48,"threads_started":1}
I20260928 22:00:40.104020 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.091866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.2.165.66:46189)
0928 22:00:40.092752 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.092758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.094444 (+ 1686us) server_negotiation.cc:407] Connection header received
0928 22:00:40.094498 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.094501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.094517 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.094583 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:00:40.094775 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.095865 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.100868 (+ 5003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.101654 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.101724 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.101796 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.103886 (+ 2090us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.103938 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":74,"threads_started":1}
I20260928 22:00:40.104912 14213 tablet_service.cc:1467] Processing CreateTablet for tablet 7447189a9081486895b3bee832142e72 (DEFAULT_TABLE table=test-table [id=bab14416136442b1b5f46387ac283277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:00:40.105280 14213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7447189a9081486895b3bee832142e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:00:40.110190 14076 tablet_service.cc:1467] Processing CreateTablet for tablet 7447189a9081486895b3bee832142e72 (DEFAULT_TABLE table=test-table [id=bab14416136442b1b5f46387ac283277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:00:40.110577 14076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7447189a9081486895b3bee832142e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:00:40.116448 14433 tablet_bootstrap.cc:492] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: Bootstrap starting.
I20260928 22:00:40.119081 14433 tablet_bootstrap.cc:654] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: Neither blocks nor log segments found. Creating new log.
I20260928 22:00:40.120405 14434 tablet_bootstrap.cc:492] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795: Bootstrap starting.
I20260928 22:00:40.121807 14434 tablet_bootstrap.cc:654] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795: Neither blocks nor log segments found. Creating new log.
I20260928 22:00:40.122284 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.091682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:35867 (local address 127.0.0.1:52226)
0928 22:00:40.096622 (+ 4940us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.096633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:40.096690 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:40.114891 (+ 18201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:40.114893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:40.114896 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:00:40.114959 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.114961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.116695 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.116697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.117896 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.117899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.122011 (+ 4112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.122012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.122103 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.122110 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:00:40.122145 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 22:00:40.122200 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4923,"thread_start_us":56,"threads_started":1}
I20260928 22:00:40.122932 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.091812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.2.165.67:35867)
0928 22:00:40.114704 (+ 22892us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.114710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.114723 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:00:40.114764 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.114766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.114783 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.115031 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:00:40.115104 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.116205 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.119320 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.120152 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.122642 (+ 2490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.122768 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.122800 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.122852 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22847,"thread_start_us":94,"threads_started":1}
I20260928 22:00:40.123541 14433 log.cc:824] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: Log is configured to *not* fsync() on all Append() calls
I20260928 22:00:40.124346 14348 tablet_service.cc:1467] Processing CreateTablet for tablet 7447189a9081486895b3bee832142e72 (DEFAULT_TABLE table=test-table [id=bab14416136442b1b5f46387ac283277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:00:40.124747 14348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7447189a9081486895b3bee832142e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:00:40.129498 14434 log.cc:824] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795: Log is configured to *not* fsync() on all Append() calls
I20260928 22:00:40.129936 14435 tablet_bootstrap.cc:492] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Bootstrap starting.
I20260928 22:00:40.130987 14435 tablet_bootstrap.cc:654] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Neither blocks nor log segments found. Creating new log.
I20260928 22:00:40.131354 14435 log.cc:824] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:00:40.132264 14433 tablet_bootstrap.cc:492] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: No bootstrap required, opened a new log
I20260928 22:00:40.132397 14433 ts_tablet_manager.cc:1397] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260928 22:00:40.134703 14433 raft_consensus.cc:348] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.135093 14433 raft_consensus.cc:374] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:00:40.135185 14433 raft_consensus.cc:729] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b669582825447e9f746b0193f94253, State: Initialized, Role: FOLLOWER
I20260928 22:00:40.135403 14433 consensus_queue.cc:260] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.135846 14433 ts_tablet_manager.cc:1428] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:00:40.136016 14279 heartbeater.cc:503] Master 127.2.165.126:39689 was elected leader, sending a full tablet report...
I20260928 22:00:40.137960 14434 tablet_bootstrap.cc:492] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795: No bootstrap required, opened a new log
I20260928 22:00:40.138100 14434 ts_tablet_manager.cc:1397] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20260928 22:00:40.141448 14434 raft_consensus.cc:348] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.141776 14434 raft_consensus.cc:374] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:00:40.141855 14434 raft_consensus.cc:729] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2880e93caa14dafaf4ede04760e9795, State: Initialized, Role: FOLLOWER
I20260928 22:00:40.142105 14434 consensus_queue.cc:260] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.142542 14434 ts_tablet_manager.cc:1428] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:00:40.142743 14142 heartbeater.cc:503] Master 127.2.165.126:39689 was elected leader, sending a full tablet report...
I20260928 22:00:40.145542 14435 tablet_bootstrap.cc:492] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: No bootstrap required, opened a new log
I20260928 22:00:40.145803 14435 ts_tablet_manager.cc:1397] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260928 22:00:40.148144 14435 raft_consensus.cc:348] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.148447 14435 raft_consensus.cc:374] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:00:40.148548 14435 raft_consensus.cc:729] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565314e9463746cab1f1c131d473c59e, State: Initialized, Role: FOLLOWER
I20260928 22:00:40.148813 14435 consensus_queue.cc:260] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.149214 14435 ts_tablet_manager.cc:1428] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:00:40.149401 14414 heartbeater.cc:503] Master 127.2.165.126:39689 was elected leader, sending a full tablet report...
I20260928 22:00:40.175416 14440 raft_consensus.cc:482] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:00:40.175580 14440 raft_consensus.cc:504] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.176013 14440 leader_election.cc:288] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44b669582825447e9f746b0193f94253 (127.2.165.66:46189), 565314e9463746cab1f1c131d473c59e (127.2.165.67:35867)
I20260928 22:00:40.181036 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.176270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:46189 (local address 127.2.165.65:57613)
0928 22:00:40.176754 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.176766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:40.176823 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:40.177465 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:40.177466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:40.177468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:00:40.177560 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.177563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.178596 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.178597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.179809 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.179814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.180723 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.180725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.180815 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.180820 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:00:40.180873 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:00:40.180930 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":106,"threads_started":1}
I20260928 22:00:40.181514 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.176927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:57613 (local address 127.2.165.66:46189)
0928 22:00:40.177146 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.177148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.177156 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:00:40.177185 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.177186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.177197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.177245 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:00:40.177707 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.178474 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.179914 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.180560 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.180646 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.180656 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.181382 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.181432 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260928 22:00:40.181802 14233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7447189a9081486895b3bee832142e72" candidate_uuid: "b2880e93caa14dafaf4ede04760e9795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b669582825447e9f746b0193f94253" is_pre_election: true
I20260928 22:00:40.181974 14233 raft_consensus.cc:2454] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2880e93caa14dafaf4ede04760e9795 in term 0.
I20260928 22:00:40.182238 14030 leader_election.cc:302] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b669582825447e9f746b0193f94253, b2880e93caa14dafaf4ede04760e9795; no voters:
I20260928 22:00:40.182389 14440 raft_consensus.cc:2781] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260928 22:00:40.182442 14143 tablet.cc:2369] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:00:40.182444 14440 raft_consensus.cc:482] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:00:40.182689 14440 raft_consensus.cc:3037] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:00:40.183470 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.176481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:35867 (local address 127.2.165.65:33977)
0928 22:00:40.177002 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.177013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:00:40.177054 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:00:40.179914 (+ 2860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:00:40.179915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:00:40.179917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:00:40.179986 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.179988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.180977 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.180978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.182351 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:00:40.182354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.183179 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.183180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:00:40.183255 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.183258 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:00:40.183326 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:00:40.183365 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":55,"threads_started":1}
I20260928 22:00:40.183492 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.176642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:33977 (local address 127.2.165.67:35867)
0928 22:00:40.177319 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.177321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.177329 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:00:40.177353 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.177354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.177365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.177412 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:00:40.180082 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.180851 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.182480 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.183020 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.183070 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.183079 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.183340 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.183389 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638}
I20260928 22:00:40.183606 14440 raft_consensus.cc:504] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.183740 14440 leader_election.cc:288] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [CANDIDATE]: Term 1 election: Requested vote from peers 44b669582825447e9f746b0193f94253 (127.2.165.66:46189), 565314e9463746cab1f1c131d473c59e (127.2.165.67:35867)
I20260928 22:00:40.183961 14233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7447189a9081486895b3bee832142e72" candidate_uuid: "b2880e93caa14dafaf4ede04760e9795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b669582825447e9f746b0193f94253"
I20260928 22:00:40.184075 14233 raft_consensus.cc:3037] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:00:40.184119 14368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7447189a9081486895b3bee832142e72" candidate_uuid: "b2880e93caa14dafaf4ede04760e9795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565314e9463746cab1f1c131d473c59e" is_pre_election: true
I20260928 22:00:40.184274 14368 raft_consensus.cc:2454] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2880e93caa14dafaf4ede04760e9795 in term 0.
I20260928 22:00:40.184415 14367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7447189a9081486895b3bee832142e72" candidate_uuid: "b2880e93caa14dafaf4ede04760e9795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565314e9463746cab1f1c131d473c59e"
I20260928 22:00:40.184520 14367 raft_consensus.cc:3037] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:00:40.185526 14233 raft_consensus.cc:2454] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2880e93caa14dafaf4ede04760e9795 in term 1.
I20260928 22:00:40.185760 14030 leader_election.cc:302] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b669582825447e9f746b0193f94253, b2880e93caa14dafaf4ede04760e9795; no voters:
I20260928 22:00:40.185874 14440 raft_consensus.cc:2781] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:00:40.185974 14367 raft_consensus.cc:2454] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2880e93caa14dafaf4ede04760e9795 in term 1.
I20260928 22:00:40.186087 14440 raft_consensus.cc:686] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [term 1 LEADER]: Becoming Leader. State: Replica: b2880e93caa14dafaf4ede04760e9795, State: Running, Role: LEADER
I20260928 22:00:40.186213 14440 consensus_queue.cc:237] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } }
I20260928 22:00:40.187156 13929 catalog_manager.cc:5495] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2880e93caa14dafaf4ede04760e9795 (127.2.165.65). New cstate: current_term: 1 leader_uuid: "b2880e93caa14dafaf4ede04760e9795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2880e93caa14dafaf4ede04760e9795" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 43153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 } health_report { overall_health: UNKNOWN } } }
Sep 28 22:00:40 dist-test-slave-3t36 krb5kdc[13454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
W20260928 22:00:40.208869 14280 tablet.cc:2369] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:00:40.219297 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.216005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.2.165.65:43153)
0928 22:00:40.216051 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:00:40.216053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:00:40.216061 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:00:40.216104 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:00:40.216105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:00:40.216130 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:00:40.216203 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:00:40.216444 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.217141 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.218204 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:00:40.218584 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:00:40.218646 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:00:40.218886 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:00:40.219026 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:00:40.219122 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:00:40.219142 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 22:00:40.219166 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:00:40.224232 14367 raft_consensus.cc:1261] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 1 FOLLOWER]: Refusing update from remote peer b2880e93caa14dafaf4ede04760e9795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:00:40.224697 14444 consensus_queue.cc:1035] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565314e9463746cab1f1c131d473c59e" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:00:40.227084 14233 raft_consensus.cc:1261] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 1 FOLLOWER]: Refusing update from remote peer b2880e93caa14dafaf4ede04760e9795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:00:40.227358 14444 consensus_queue.cc:1035] T 7447189a9081486895b3bee832142e72 P b2880e93caa14dafaf4ede04760e9795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44b669582825447e9f746b0193f94253" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:00:40.237156 14448 mvcc.cc:205] Tried to move back new op lower bound from 7334432113546657792 to 7334432113403371520. Current Snapshot: MvccSnapshot[applied={T|T < 7334432113546657792}]
I20260928 22:00:40.237213 14447 mvcc.cc:205] Tried to move back new op lower bound from 7334432113546657792 to 7334432113403371520. Current Snapshot: MvccSnapshot[applied={T|T < 7334432113546657792}]
I20260928 22:00:40.238621 14450 mvcc.cc:205] Tried to move back new op lower bound from 7334432113546657792 to 7334432113403371520. Current Snapshot: MvccSnapshot[applied={T|T < 7334432113546657792}]
I20260928 22:00:40.245354 13929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:00:40.245491 13929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:00:40.246449 13929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f230aef4ec3346fcbe7ddbe50fa69e1c: Sending DeleteTablet for 3 replicas of tablet 7447189a9081486895b3bee832142e72
I20260928 22:00:40.246997 14076 tablet_service.cc:1514] Processing DeleteTablet for tablet 7447189a9081486895b3bee832142e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:00:40 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:37316
I20260928 22:00:40.247327 14213 tablet_service.cc:1514] Processing DeleteTablet for tablet 7447189a9081486895b3bee832142e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:00:40 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:35544
I20260928 22:00:40.247593 14348 tablet_service.cc:1514] Processing DeleteTablet for tablet 7447189a9081486895b3bee832142e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:00:40 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:52226
I20260928 22:00:40.247967 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 14011
I20260928 22:00:40.250332 14457 tablet_replica.cc:330] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: stopping tablet replica
I20260928 22:00:40.251686 14457 raft_consensus.cc:2229] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:00:40.252102 14457 raft_consensus.cc:2258] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:00:40.252442 14458 tablet_replica.cc:330] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: stopping tablet replica
I20260928 22:00:40.252960 14457 ts_tablet_manager.cc:1905] T 7447189a9081486895b3bee832142e72 P 44b669582825447e9f746b0193f94253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:00:40.253140 14458 raft_consensus.cc:2229] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:00:40.253297 14458 raft_consensus.cc:2258] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:00:40.255357 14458 ts_tablet_manager.cc:1905] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:00:40.257035 13914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:43153 (error 108)
I20260928 22:00:40.257644 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 14148
I20260928 22:00:40.259078 14458 ts_tablet_manager.cc:1918] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:00:40.259161 14458 log.cc:1196] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/7447189a9081486895b3bee832142e72
I20260928 22:00:40.261315 14458 ts_tablet_manager.cc:1939] T 7447189a9081486895b3bee832142e72 P 565314e9463746cab1f1c131d473c59e: Deleting consensus metadata
I20260928 22:00:40.261940 13917 catalog_manager.cc:4841] TS 565314e9463746cab1f1c131d473c59e (127.2.165.67:35867): tablet 7447189a9081486895b3bee832142e72 (table test-table [id=bab14416136442b1b5f46387ac283277]) successfully deleted
I20260928 22:00:40.262526 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.262388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:43153 (local address 127.0.0.1:38752)
0928 22:00:40.262443 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.262483 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:00:40.262741 13914 catalog_manager.cc:4584] TS b2880e93caa14dafaf4ede04760e9795 (127.2.165.65:43153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7447189a9081486895b3bee832142e72: Network error: Client connection negotiation failed: client connection to 127.2.165.65:43153: connect: Connection refused (error 111)
I20260928 22:00:40.263460 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:00:40.263373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:46189 (local address 127.0.0.1:46944)
0928 22:00:40.263413 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:00:40.263434 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.66:46189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 22:00:40.263557 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 14285
I20260928 22:00:40.268273 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 13898
I20260928 22:00:40.273037 2709 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:00:40.619319 2709 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:00:40.619472 2709 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:00:40.964949 2709 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:00:40.965 UTC [13484] LOG: received smart shutdown request
2026-09-28 22:00:40.969 UTC [13484] LOG: background worker "logical replication launcher" (PID 13493) exited with exit code 1
2026-09-28 22:00:40.969 UTC [13488] LOG: shutting down
2026-09-28 22:00:41.182 UTC [13484] LOG: database system is shut down
2026-09-28T22:00:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49372 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:00:41 dist-test-slave-3t36 krb5kdc[14468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:00:41 dist-test-slave-3t36 krb5kdc[14468](info): set up 2 sockets
Sep 28 22:00:41 dist-test-slave-3t36 krb5kdc[14468](info): commencing operation
krb5kdc: starting...
W20260928 22:00:42.344837 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:00:42 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:00:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/rangeradmin_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/rangeradmin_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/rangerlookup_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/rangerlookup_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
I20260928 22:00:42.533265 2709 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjhUuNU/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/postgres -l logfile start
2026-09-28 22:00:45.441 UTC [14498] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 22:00:45.441 UTC [14498] LOG: listening on IPv4 address "127.2.165.84", port 53749
2026-09-28 22:00:45.445 UTC [14498] LOG: listening on Unix socket "/tmp/.s.PGSQL.53749"
2026-09-28 22:00:45.481 UTC [14501] LOG: database system was shut down at 2026-09-28 22:00:43 UTC
2026-09-28 22:00:45.486 UTC [14498] LOG: database system is ready to accept connections
I20260928 22:00:46.456570 2709 mini_postgres.cc:96] Postgres bound to 53749
2026-09-28 22:00:46.462 UTC [14509] FATAL: database "slave" does not exist
127.2.165.84:53749 - accepting connections
I20260928 22:00:46.463153 2709 mini_ranger.cc:161] Starting Ranger...
I20260928 22:00:46.479316 2709 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 22:00:47.047559 2709 mini_ranger.cc:87] Created ranger Postgres database
I20260928 22:00:47.047659 2709 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-admin
2026-09-28 22:00:47,339 [I] DB FLAVOR :POSTGRES
2026-09-28 22:00:47,340 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:00:47,340 [I] Checking connection..
2026-09-28 22:00:47,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 22:00:47,617 [I] Checking connection passed.
2026-09-28 22:00:47,617 [I] --------- Verifying version history table ---------
2026-09-28 22:00:47,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:00:47,861 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:00:47,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:00:48,123 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:00:48,124 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 22:00:48,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 22:00:48.388 UTC [14573] WARNING: there is no transaction in progress
2026-09-28 22:00:48,394 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 22:00:48,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:00:48,651 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 22:00:48,652 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:00:48,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 22:00:48,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 22:00:49,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3t36.c.gcp-upstream.internal','N') ;"
2026-09-28 22:00:49,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:00:49,670 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:00:49,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:00:49,919 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:00:49,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:00:50,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:00:50,469 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:00:50,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:00:51.565 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.585 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.604 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.622 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.641 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.661 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.722 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.730 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.746 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.757 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.769 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.785 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.796 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.804 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.813 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:51.821 UTC [14681] WARNING: there is no transaction in progress
2026-09-28 22:00:52,619 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:00:52,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:00:52,856 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:00:52,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:00:53,115 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:00:53,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:00:53,376 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:00:53,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:00:53,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:00:53,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3t36.c.gcp-upstream.internal';"
2026-09-28 22:00:54,124 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:00:54,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:00:54,379 [I] DB_PATCHES have already been applied
I20260928 22:00:54.385406 2709 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjhUuNU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:00:54.385473 2709 mini_ranger.cc:193] Using host: 127.2.165.84
I20260928 22:00:54.388571 2709 mini_ranger.cc:227] Ranger admin URL: http://127.2.165.84:41275
Sep 28, 2026 10:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:00:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:00:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:00:55.566406 2709 mini_ranger.cc:160] Time spent starting Ranger: real 9.103s user 0.000s sys 0.010s
Sep 28, 2026 10:01:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:01:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 10:01:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:01:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:01:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:01:17.476933 2709 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.165.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.165.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/rangerkms_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.165.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/rangerkms_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.165.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.165.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.165.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/HTTP_127.2.165.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:01:17 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:01:17.563393 2709 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:01:17.572942 2709 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:01:18.594921 2709 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:01:18.595009 2709 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms
I20260928 22:01:18.595031 2709 mini_ranger_kms.cc:227] Using postgres at 127.2.165.84:53749
2026-09-28 22:01:18,757 [I] DB FLAVOR :POSTGRES
2026-09-28 22:01:18,757 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:01:18,757 [I] Checking connection
2026-09-28 22:01:18,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:01:18,996 [I] connection success
2026-09-28 22:01:18,996 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:01:18,996 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:01:18,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:01:19,251 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:01:19,251 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:01:19,252 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:01:19,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.165.84:53749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:01:19,533 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:01:20.001003 2709 mini_ranger_kms.cc:310] Created kms service
I20260928 22:01:20.229362 2709 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:01:20.273949 2709 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:01:20 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:20.669466 2709 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:01:20.669620 2709 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjhUuNU/build/release/bin/postgresql.jar:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjhUuNU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:01:20.669642 2709 mini_ranger_kms.cc:240] Using host: 127.2.165.84
I20260928 22:01:20.673177 2709 mini_ranger_kms.cc:276] Ranger KMS PID: 14861
I20260928 22:01:20.673267 2709 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.165.84:59211
Sep 28, 2026 10:01:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:01:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:01:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:01:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:01:22.943522 2709 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.380s user 0.010s sys 0.135s
I20260928 22:01:22.943604 2709 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:01:22.943642 2709 mini_ranger_kms.cc:401] 127.2.165.84:59211/kms/v1/keys
Sep 28 22:01:23 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.165.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:01:23 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.165.84@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
Sep 28 22:01:23 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.165.84@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
Sep 28 22:01:26 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:01:26.609714 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46331
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:46331
--ranger_config_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:26.718391 14912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:26.718644 14912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:26.718691 14912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:26.719885 14912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:26.719954 14912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:26.719981 14912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:26.720016 14912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:26.720046 14912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:26.721748 14912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.165.84:59211/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:46331
--ranger_config_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:26.722026 14912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:26.722252 14912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:26.727360 14918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:26.727347 14920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:26.727361 14917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:26.727612 14912 server_base.cc:1034] running on GCE node
I20260928 22:01:26.727937 14912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:26.728279 14912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:26.729480 14912 hybrid_clock.cc:648] HybridClock initialized: now 1790632886729458 us; error 41 us; skew 500 ppm
Sep 28 22:01:26 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632886, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:26.732089 14912 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:26.732604 14912 webserver.cc:460] Webserver started at http://127.2.165.126:34909/ using document root <none> and password file <none>
I20260928 22:01:26.732882 14912 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:26.732946 14912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:26.733117 14912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:01:26 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632886, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:26.781802 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "6b0c62cf757b40e8ac5524e51cd677f5"
format_stamp: "Formatted at 2026-09-28 22:01:26 on dist-test-slave-3t36"
server_key: "9be02b9b3f27051afce0cb78127bba2d"
server_key_iv: "e76ad4abddcc1bc1350425741e2fbf2c"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:26.782238 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "6b0c62cf757b40e8ac5524e51cd677f5"
format_stamp: "Formatted at 2026-09-28 22:01:26 on dist-test-slave-3t36"
server_key: "9be02b9b3f27051afce0cb78127bba2d"
server_key_iv: "e76ad4abddcc1bc1350425741e2fbf2c"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:26.783676 14912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:01:26.795413 14927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:26.795655 14912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:26.795758 14912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "6b0c62cf757b40e8ac5524e51cd677f5"
format_stamp: "Formatted at 2026-09-28 22:01:26 on dist-test-slave-3t36"
server_key: "9be02b9b3f27051afce0cb78127bba2d"
server_key_iv: "e76ad4abddcc1bc1350425741e2fbf2c"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:26.795888 14912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:26.807137 14912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:26.809427 14912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:26.809604 14912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:26.814597 14912 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:46331
I20260928 22:01:26.814620 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:46331 every 8 connection(s)
I20260928 22:01:26.815097 14912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:26.815827 14980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:26.816706 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 14912
I20260928 22:01:26.816826 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:01:26.817044 2709 external_mini_cluster.cc:1428] Setting key b1ca01b1150d2f30d6cae15238519007
I20260928 22:01:26.818845 14980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Bootstrap starting.
I20260928 22:01:26.819643 14980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:26.819989 14980 log.cc:824] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:26.820791 14980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: No bootstrap required, opened a new log
I20260928 22:01:26.823057 14980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:26.823344 14980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:26.823410 14980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0c62cf757b40e8ac5524e51cd677f5, State: Initialized, Role: FOLLOWER
I20260928 22:01:26.823902 14980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:26.824009 14980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:01:26.824081 14980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:01:26.824144 14980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:26.824973 14980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:26.825104 14980 leader_election.cc:302] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b0c62cf757b40e8ac5524e51cd677f5; no voters:
I20260928 22:01:26.825306 14980 leader_election.cc:288] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:01:26.825343 14985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 22:01:26 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:26.825541 14985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b0c62cf757b40e8ac5524e51cd677f5, State: Running, Role: LEADER
I20260928 22:01:26.825623 14985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:26.825695 14980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:26.825973 14985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } } }
I20260928 22:01:26.826059 14985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:26.826202 14985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0c62cf757b40e8ac5524e51cd677f5. Latest consensus state: current_term: 1 leader_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } } }
I20260928 22:01:26.826360 14985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:26.827219 14980 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:01:26.831507 14990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:26.832491 14990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:26.833859 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:26.817793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.2.165.126:46331)
0928 22:01:26.817993 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:26.818005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:26.818047 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:01:26.818775 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:26.818786 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:26.818866 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:26.819059 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:26.819871 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:26.821367 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:26.822488 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:26.823100 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:26.823209 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:26.826223 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:26.826228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:26.826228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:26.826243 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:26.830984 (+ 4741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:26.831599 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:26.831603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:26.831603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:26.831738 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:26.832014 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:26.832015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:26.832015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:26.832575 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:26.832899 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:26.833464 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 22:01:26.833627 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260928 22:01:26.835486 14990 catalog_manager.cc:1347] Generated new cluster ID: 5ae48c240bf14d548a84131db62c712d
I20260928 22:01:26.835538 14990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:01:26.848770 14990 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:01:26.850085 14990 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:01:26.855588 14990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Generated new TSK 0
I20260928 22:01:26.855906 14990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 22:01:27.633590 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46331
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:27.731467 15025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:27.731717 15025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:27.731760 15025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:27.733018 15025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:27.733083 15025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:27.733129 15025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:01:27.734722 15025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.165.84:59211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:27.734997 15025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:27.735236 15025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:27.741227 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:27.741230 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:27.741655 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:27.741920 15025 server_base.cc:1034] running on GCE node
I20260928 22:01:27.742136 15025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:27.742419 15025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:27.743611 15025 hybrid_clock.cc:648] HybridClock initialized: now 1790632887743577 us; error 44 us; skew 500 ppm
Sep 28 22:01:27 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632887, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:27.746903 15025 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:01:27.747380 15025 webserver.cc:460] Webserver started at http://127.2.165.65:36131/ using document root <none> and password file <none>
I20260928 22:01:27.747628 15025 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:27.747697 15025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:27.747852 15025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:01:27 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632887, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:27.762734 15025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "ff68b0817fb8471e854b91f5c131437e"
format_stamp: "Formatted at 2026-09-28 22:01:27 on dist-test-slave-3t36"
server_key: "dd69fa912ff6354687895fb8b1af62ac"
server_key_iv: "0b604c8efe9ff033cb66707bf6b9305a"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:27.763213 15025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "ff68b0817fb8471e854b91f5c131437e"
format_stamp: "Formatted at 2026-09-28 22:01:27 on dist-test-slave-3t36"
server_key: "dd69fa912ff6354687895fb8b1af62ac"
server_key_iv: "0b604c8efe9ff033cb66707bf6b9305a"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:27.764739 15025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:01:27.777097 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:27.777371 15025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:27.777477 15025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "ff68b0817fb8471e854b91f5c131437e"
format_stamp: "Formatted at 2026-09-28 22:01:27 on dist-test-slave-3t36"
server_key: "dd69fa912ff6354687895fb8b1af62ac"
server_key_iv: "0b604c8efe9ff033cb66707bf6b9305a"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:27.777571 15025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:27.794915 15025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:27.799042 15025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:27.799315 15025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:27.799676 15025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:27.800180 15025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:27.800300 15025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:27.800419 15025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:27.800487 15025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:27.807338 15025 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:38761
I20260928 22:01:27.807900 15025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:01:27.811466 15155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:38761 every 8 connection(s)
I20260928 22:01:27.811632 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 15025
I20260928 22:01:27.811723 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:01:27.811949 2709 external_mini_cluster.cc:1428] Setting key f743d0bb05dc1f6cada375929b854886
Sep 28 22:01:27 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632887, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
I20260928 22:01:27.834216 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:27.817765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:56423 (local address 127.2.165.126:46331)
0928 22:01:27.820647 (+ 2882us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:27.820650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:27.822281 (+ 1631us) server_negotiation.cc:407] Connection header received
0928 22:01:27.823106 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:27.823108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:27.823123 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:27.823174 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:27.824165 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:27.825086 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:27.826851 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:27.827197 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:27.827251 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:27.830283 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:27.830285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:27.830286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:27.830295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:27.832325 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:27.832923 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:27.832924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:27.832925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:27.832996 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:27.833319 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:27.833320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:27.833321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:27.833481 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:27.833542 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:27.833990 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 22:01:27.834090 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2846,"thread_start_us":100,"threads_started":1}
I20260928 22:01:27.834803 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:27.817736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46331 (local address 127.2.165.65:56423)
0928 22:01:27.822105 (+ 4369us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:27.822170 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:27.822952 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:27.823393 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:27.823398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:27.823696 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:27.824033 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:27.824041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:27.825219 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:27.825220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:27.826672 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:27.826676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:27.827368 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:27.827369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:27.827529 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:27.828233 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:27.828245 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:27.830152 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:27.832435 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:27.832443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:27.832447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:27.832815 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:27.833081 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:27.833082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:27.833082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:27.833227 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:27.833582 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:27.833589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:27.833795 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 22:01:27.834337 (+ 542us) client_negotiation.cc:240] Negotiation successful
0928 22:01:27.834523 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4312,"thread_start_us":97,"threads_started":1}
I20260928 22:01:27.835474 15156 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46331
I20260928 22:01:27.835580 15156 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:27.835801 15156 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:27.836421 14944 ts_manager.cc:194] Registered new tserver with Master: ff68b0817fb8471e854b91f5c131437e (127.2.165.65:38761)
I20260928 22:01:27.837525 14944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:56423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 22:01:27.861977 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46331
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:27.978420 15162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:27.978789 15162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:27.978864 15162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:27.980062 15162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:27.980129 15162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:27.980242 15162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:01:27.982427 15162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.165.84:59211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:27.982720 15162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:27.982973 15162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:27.987979 15169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:27.988097 15172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:27.988283 15170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:27.989219 15162 server_base.cc:1034] running on GCE node
I20260928 22:01:27.989440 15162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:27.989675 15162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:27.990862 15162 hybrid_clock.cc:648] HybridClock initialized: now 1790632887990787 us; error 86 us; skew 500 ppm
Sep 28 22:01:27 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632887, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:27.994019 15162 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:01:27.994589 15162 webserver.cc:460] Webserver started at http://127.2.165.66:32973/ using document root <none> and password file <none>
I20260928 22:01:27.994851 15162 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:27.994921 15162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:27.995057 15162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:01:27 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632887, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:28.028801 15162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "eddfda17bb7549bf80de8451d80b5fc1"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "a34a254bcd8508e8ff172f3cff643356"
server_key_iv: "35108960ca9d1df998bc6e71a8301d26"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.029369 15162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "eddfda17bb7549bf80de8451d80b5fc1"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "a34a254bcd8508e8ff172f3cff643356"
server_key_iv: "35108960ca9d1df998bc6e71a8301d26"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.034592 15162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260928 22:01:28.049764 15179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.050027 15162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:01:28.050113 15162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "eddfda17bb7549bf80de8451d80b5fc1"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "a34a254bcd8508e8ff172f3cff643356"
server_key_iv: "35108960ca9d1df998bc6e71a8301d26"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.050206 15162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:28.062084 15162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:28.064564 15162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:28.064741 15162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:28.065004 15162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:28.065429 15162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:28.065507 15162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.065564 15162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:28.065606 15162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.073024 15162 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:35573
I20260928 22:01:28.073585 15162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:01:28.075516 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 15162
I20260928 22:01:28.075632 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:01:28.075861 2709 external_mini_cluster.cc:1428] Setting key 89600f61e7af22c2d53d0516d54e197c
I20260928 22:01:28.080873 15292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:35573 every 8 connection(s)
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632887, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:28.096195 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:28.074460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:46955 (local address 127.2.165.126:46331)
0928 22:01:28.074510 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:28.074513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:28.085416 (+ 10903us) server_negotiation.cc:407] Connection header received
0928 22:01:28.085772 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:28.085773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:28.085822 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:28.085923 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:28.087215 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.088060 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.089644 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.089935 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.089981 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:28.092780 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:28.092782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:28.092782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:28.092791 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:28.094565 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:28.095069 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:28.095070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:28.095070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:28.095126 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:28.095411 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:28.095412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:28.095412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:28.095549 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:28.095600 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:28.095958 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 22:01:28.096056 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:01:28.097018 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:28.074436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46331 (local address 127.2.165.66:46955)
0928 22:01:28.084596 (+ 10160us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:28.084704 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:28.085583 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:28.086162 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:28.086169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:28.086551 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:28.087030 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:28.087039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.088189 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.088190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:28.089538 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:28.089541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.090090 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.090091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:28.090233 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:28.090888 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:28.090899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:28.092627 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:28.094664 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:28.094671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:28.094674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:28.094981 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:28.095191 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:28.095192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:28.095192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:28.095336 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:28.095632 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:28.095637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:28.095803 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 22:01:28.096586 (+ 783us) client_negotiation.cc:240] Negotiation successful
0928 22:01:28.096778 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10090,"thread_start_us":85,"threads_started":1}
I20260928 22:01:28.097823 15293 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46331
I20260928 22:01:28.097918 15293 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:28.098099 15293 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:28.098533 14944 ts_manager.cc:194] Registered new tserver with Master: eddfda17bb7549bf80de8451d80b5fc1 (127.2.165.66:35573)
I20260928 22:01:28.098995 14944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:46955
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 22:01:28.128153 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46331
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:28.255198 15299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:28.255553 15299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:28.255636 15299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:28.257519 15299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:28.257795 15299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:28.257907 15299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:01:28.260306 15299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.165.84:59211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:28.260902 15299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:28.261294 15299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:28.269434 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.269829 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.270646 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:28.271284 15299 server_base.cc:1034] running on GCE node
I20260928 22:01:28.271528 15299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:28.271824 15299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:28.272992 15299 hybrid_clock.cc:648] HybridClock initialized: now 1790632888272969 us; error 36 us; skew 500 ppm
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:28.276840 15299 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:01:28.277485 15299 webserver.cc:460] Webserver started at http://127.2.165.67:44407/ using document root <none> and password file <none>
I20260928 22:01:28.278137 15299 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:28.278291 15299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:28.278499 15299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:28.296875 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "45a09043c6054eb8bfccf5edd0aa1fb6"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "8f16c8f4986239e7e7908674fe29a0a1"
server_key_iv: "16125bf50f9427d3a6d00e7f270df75d"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.297515 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "45a09043c6054eb8bfccf5edd0aa1fb6"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "8f16c8f4986239e7e7908674fe29a0a1"
server_key_iv: "16125bf50f9427d3a6d00e7f270df75d"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.300269 15299 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 22:01:28.317464 15314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.317855 15299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 22:01:28.317958 15299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "45a09043c6054eb8bfccf5edd0aa1fb6"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "8f16c8f4986239e7e7908674fe29a0a1"
server_key_iv: "16125bf50f9427d3a6d00e7f270df75d"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.318058 15299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:28.334367 15299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:28.337028 15299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:28.337201 15299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:28.337468 15299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:28.337880 15299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:28.337939 15299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.337970 15299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:28.338059 15299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.346119 15299 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:37637
I20260928 22:01:28.346689 15299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:01:28.347525 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:37637 every 8 connection(s)
I20260928 22:01:28.350968 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 15299
I20260928 22:01:28.351094 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:01:28.351337 2709 external_mini_cluster.cc:1428] Setting key a53ce2deb24813cdcdbaac5ed4038a8b
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:28.378144 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:28.348509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:40755 (local address 127.2.165.126:46331)
0928 22:01:28.348577 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:28.348580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:28.350158 (+ 1578us) server_negotiation.cc:407] Connection header received
0928 22:01:28.350979 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:28.350981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:28.350995 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:28.351043 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:28.351948 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.352758 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.355063 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.355371 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.355448 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:28.359127 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:28.359129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:28.359129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:28.359139 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:28.375088 (+ 15949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:28.375735 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:28.375737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:28.375738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:28.375802 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:28.376182 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:28.376183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:28.376183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:28.377419 (+ 1236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:28.377476 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:28.377944 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 22:01:28.378044 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:01:28.378720 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:28.348470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46331 (local address 127.2.165.67:40755)
0928 22:01:28.349962 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:28.350019 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:28.350692 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:28.351174 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:28.351179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:28.351529 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:28.351812 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:28.351818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.352855 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.352856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:28.354900 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:28.354905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.355505 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.355507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:28.355712 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:28.356876 (+ 1164us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:28.356891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:28.358986 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:28.375241 (+ 16255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:28.375256 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:28.375260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:28.375636 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:28.375956 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:28.375956 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:28.375957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:28.376106 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:28.377531 (+ 1425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:28.377540 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:28.377733 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 22:01:28.378284 (+ 551us) client_negotiation.cc:240] Negotiation successful
0928 22:01:28.378473 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1440,"thread_start_us":104,"threads_started":1}
I20260928 22:01:28.379136 15428 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46331
I20260928 22:01:28.379222 15428 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:28.379410 15428 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:28.379725 14944 ts_manager.cc:194] Registered new tserver with Master: 45a09043c6054eb8bfccf5edd0aa1fb6 (127.2.165.67:37637)
I20260928 22:01:28.380213 14944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:40755
I20260928 22:01:28.388214 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:01:28.389189 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 15025
I20260928 22:01:28.395460 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 15162
I20260928 22:01:28.403990 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 15299
I20260928 22:01:28.411100 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 14912
I20260928 22:01:28.416433 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46331
--webserver_interface=127.2.165.126
--webserver_port=34909
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:46331
--ranger_config_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:28.505220 15431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:28.505472 15431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:28.505509 15431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:28.506719 15431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:28.506790 15431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:28.506814 15431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:28.506845 15431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:28.506891 15431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:28.508517 15431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:46331
--ranger_config_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=34909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:28.508841 15431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:28.509083 15431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:28.514593 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.514604 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.514660 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:28.515892 15431 server_base.cc:1034] running on GCE node
I20260928 22:01:28.516213 15431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:28.516628 15431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:28.517838 15431 hybrid_clock.cc:648] HybridClock initialized: now 1790632888517813 us; error 43 us; skew 500 ppm
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:28.520964 15431 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:28.521530 15431 webserver.cc:460] Webserver started at http://127.2.165.126:34909/ using document root <none> and password file <none>
I20260928 22:01:28.521790 15431 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:28.521867 15431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:28.574585 15431 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:01:28.575529 15447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.575734 15431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:28.575834 15431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "6b0c62cf757b40e8ac5524e51cd677f5"
format_stamp: "Formatted at 2026-09-28 22:01:26 on dist-test-slave-3t36"
server_key: "9be02b9b3f27051afce0cb78127bba2d"
server_key_iv: "e76ad4abddcc1bc1350425741e2fbf2c"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.576277 15431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:28.603809 15431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:28.606240 15431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:28.606415 15431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:28.611706 15431 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:46331
I20260928 22:01:28.611829 15499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:46331 every 8 connection(s)
I20260928 22:01:28.612231 15431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:28.613632 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 15431
I20260928 22:01:28.613776 15500 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:01:28.613955 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:38761
--local_ip_for_outbound_sockets=127.2.165.65
--tserver_master_addrs=127.2.165.126:46331
--webserver_port=36131
--webserver_interface=127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:28.614845 15500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Bootstrap starting.
I20260928 22:01:28.623238 15500 log.cc:824] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:28.625130 15500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:01:28.625497 15500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Bootstrap complete.
I20260928 22:01:28.627898 15500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:28.628217 15500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0c62cf757b40e8ac5524e51cd677f5, State: Initialized, Role: FOLLOWER
I20260928 22:01:28.628448 15500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:28.628535 15500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:01:28.628600 15500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:01:28.628695 15500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:01:28.629762 15500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:28.629885 15500 leader_election.cc:302] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b0c62cf757b40e8ac5524e51cd677f5; no voters:
I20260928 22:01:28.630115 15500 leader_election.cc:288] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:01:28.630152 15504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:01:28.630364 15504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [term 2 LEADER]: Becoming Leader. State: Replica: 6b0c62cf757b40e8ac5524e51cd677f5, State: Running, Role: LEADER
I20260928 22:01:28.630476 15500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:28.630535 15504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } }
I20260928 22:01:28.630853 15504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } } }
I20260928 22:01:28.631001 15504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:28.631184 15504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0c62cf757b40e8ac5524e51cd677f5. Latest consensus state: current_term: 2 leader_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0c62cf757b40e8ac5524e51cd677f5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46331 } } }
I20260928 22:01:28.631309 15504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:28.631451 15512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:28.632018 15512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:28.632172 15512 catalog_manager.cc:1259] Loaded cluster ID: 5ae48c240bf14d548a84131db62c712d
I20260928 22:01:28.632458 15512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:01:28.632467 15500 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:01:28.637430 15512 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:01:28.637898 15512 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Loaded TSK: 0
I20260928 22:01:28.638237 15512 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:01:28.736380 15502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:28.736804 15502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:28.736918 15502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:28.738444 15502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:28.738528 15502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:28.738575 15502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:01:28.740209 15502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:38761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=36131
--tserver_master_addrs=127.2.165.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:28.740496 15502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:28.740813 15502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:28.745958 15540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.745934 15538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.746346 15537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:28.746556 15502 server_base.cc:1034] running on GCE node
I20260928 22:01:28.746953 15502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:28.747223 15502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:28.748428 15502 hybrid_clock.cc:648] HybridClock initialized: now 1790632888748406 us; error 37 us; skew 500 ppm
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:28.751777 15502 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:01:28.752339 15502 webserver.cc:460] Webserver started at http://127.2.165.65:36131/ using document root <none> and password file <none>
I20260928 22:01:28.752610 15502 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:28.752707 15502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:28.777184 15502 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 22:01:28.779145 15548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.779362 15502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:01:28.779471 15502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "ff68b0817fb8471e854b91f5c131437e"
format_stamp: "Formatted at 2026-09-28 22:01:27 on dist-test-slave-3t36"
server_key: "dd69fa912ff6354687895fb8b1af62ac"
server_key_iv: "0b604c8efe9ff033cb66707bf6b9305a"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.780083 15502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:28.797699 15502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:28.800029 15502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:28.800230 15502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:28.800493 15502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:28.800930 15502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:28.801020 15502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.801086 15502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:28.801117 15502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.808902 15502 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:38761
I20260928 22:01:28.809305 15502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:01:28.809854 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:38761 every 8 connection(s)
I20260928 22:01:28.811977 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 15502
I20260928 22:01:28.812307 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:35573
--local_ip_for_outbound_sockets=127.2.165.66
--tserver_master_addrs=127.2.165.126:46331
--webserver_port=32973
--webserver_interface=127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:28.824322 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:28.810792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:43205 (local address 127.2.165.126:46331)
0928 22:01:28.810998 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:28.811006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:28.811058 (+ 52us) server_negotiation.cc:407] Connection header received
0928 22:01:28.811787 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:28.811795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:28.811855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:28.811993 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:28.812571 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.814041 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.815752 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.816444 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.816574 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:28.819545 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:28.819551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:28.819551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:28.819564 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:28.821958 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:28.822562 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:28.822566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:28.822567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:28.822689 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:28.823048 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:28.823049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:28.823050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:28.823382 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:28.823469 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:28.823908 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 22:01:28.824040 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260928 22:01:28.824340 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:28.810717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46331 (local address 127.2.165.65:43205)
0928 22:01:28.810915 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:28.810968 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:28.811470 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:28.812034 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:28.812038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:28.812246 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:28.812445 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:28.812450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.814157 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.814158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:28.815646 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:28.815650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:28.816586 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:28.816587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:28.816823 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:28.817504 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:28.817524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:28.819404 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:28.822056 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:28.822066 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:28.822069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:28.822452 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:28.822788 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:28.822789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:28.822790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:28.822960 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:28.823471 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:28.823478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:28.823673 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 22:01:28.823830 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 22:01:28.824033 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260928 22:01:28.825531 15662 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46331
I20260928 22:01:28.825639 15662 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:28.825829 15662 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
W20260928 22:01:28.826788 15662 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:46331 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:01:28.826859 15662 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:28.826923 15662 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:28.827425 15662 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:28.827498 15662 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
W20260928 22:01:28.827889 15662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:01:28.933431 15666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:28.933765 15666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:28.933823 15666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:28.935702 15666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:28.935801 15666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:28.935864 15666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:01:28.938216 15666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:35573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=32973
--tserver_master_addrs=127.2.165.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:28.938616 15666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:28.938910 15666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:28.944941 15674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.944998 15672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:28.945004 15671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:28.945338 15666 server_base.cc:1034] running on GCE node
I20260928 22:01:28.945574 15666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:28.945881 15666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:28.947059 15666 hybrid_clock.cc:648] HybridClock initialized: now 1790632888947033 us; error 39 us; skew 500 ppm
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:28.950052 15666 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:01:28.950577 15666 webserver.cc:460] Webserver started at http://127.2.165.66:32973/ using document root <none> and password file <none>
I20260928 22:01:28.950840 15666 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:28.950927 15666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:01:28 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:28.972304 15666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:28.973296 15682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.973489 15666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:28.973592 15666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "eddfda17bb7549bf80de8451d80b5fc1"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "a34a254bcd8508e8ff172f3cff643356"
server_key_iv: "35108960ca9d1df998bc6e71a8301d26"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:28.974007 15666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:28.991218 15666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:28.993847 15666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:28.994025 15666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:28.994309 15666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:28.994721 15666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:28.994776 15666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:28.994809 15666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:28.994827 15666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:29.002719 15666 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:35573
I20260928 22:01:29.003173 15666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:01:29.003677 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:35573 every 8 connection(s)
I20260928 22:01:29.005544 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 15666
I20260928 22:01:29.005856 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.165.84:59211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:37637
--local_ip_for_outbound_sockets=127.2.165.67
--tserver_master_addrs=127.2.165.126:46331
--webserver_port=44407
--webserver_interface=127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:01:29 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:29.014792 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:29.004376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:60159 (local address 127.2.165.126:46331)
0928 22:01:29.004428 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:29.004431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:29.004911 (+ 480us) server_negotiation.cc:407] Connection header received
0928 22:01:29.005503 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:29.005505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:29.005524 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:29.005558 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:29.006451 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.007238 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.008347 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.008571 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.008665 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:29.011621 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:29.011623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:29.011623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:29.011631 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:29.013307 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:29.013747 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:29.013749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:29.013749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:29.013813 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:29.014100 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:29.014101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:29.014101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:29.014228 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:29.014307 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:29.014608 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 22:01:29.014698 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:01:29.015003 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:29.004389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46331 (local address 127.2.165.66:60159)
0928 22:01:29.004777 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:29.004820 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:29.005356 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:29.005776 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:29.005780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:29.006026 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:29.006325 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:29.006332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.007378 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.007380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:29.008250 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:29.008252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.008735 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.008737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:29.008870 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:29.009447 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:29.009456 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:29.011503 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:29.013400 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:29.013407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:29.013410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:29.013653 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:29.013899 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:29.013899 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:29.013900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:29.014025 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:29.014297 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:29.014304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:29.014464 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 22:01:29.014575 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:01:29.014737 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":77,"threads_started":1}
I20260928 22:01:29.015570 15796 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46331
I20260928 22:01:29.015702 15796 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:29.015895 15796 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
W20260928 22:01:29.016526 15796 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:46331 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:01:29.016618 15796 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:29.016712 15796 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:29.017037 15796 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:29.017117 15796 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
W20260928 22:01:29.017406 15796 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:01:29.133575 15799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:29.133937 15799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:29.134012 15799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:29.136266 15799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:29.136344 15799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:29.136391 15799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:01:29.139019 15799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.165.84:59211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:37637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=44407
--tserver_master_addrs=127.2.165.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:29.139333 15799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:29.139680 15799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:29.146940 15805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:29.147259 15804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:29.147605 15807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:29.148486 15799 server_base.cc:1034] running on GCE node
I20260928 22:01:29.148767 15799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:29.149070 15799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:29.150266 15799 hybrid_clock.cc:648] HybridClock initialized: now 1790632889150241 us; error 37 us; skew 500 ppm
Sep 28 22:01:29 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:29.153748 15799 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:01:29.154325 15799 webserver.cc:460] Webserver started at http://127.2.165.67:44407/ using document root <none> and password file <none>
I20260928 22:01:29.154707 15799 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:29.154829 15799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:01:29 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:29.176628 15799 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:01:29.177842 15816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:29.178071 15799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:01:29.178189 15799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "45a09043c6054eb8bfccf5edd0aa1fb6"
format_stamp: "Formatted at 2026-09-28 22:01:28 on dist-test-slave-3t36"
server_key: "8f16c8f4986239e7e7908674fe29a0a1"
server_key_iv: "16125bf50f9427d3a6d00e7f270df75d"
server_key_version: "kuduclusterkey@0"
I20260928 22:01:29.178747 15799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:29.211463 15799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:29.213919 15799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:29.214073 15799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:29.214309 15799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:29.214700 15799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:29.214745 15799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:29.214774 15799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:29.214792 15799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:29.221630 15799 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:37637
I20260928 22:01:29.221786 15929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:37637 every 8 connection(s)
I20260928 22:01:29.222067 15799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:01:29.227126 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 15799
Sep 28 22:01:29 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:29.232872 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:29.222906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:58593 (local address 127.2.165.126:46331)
0928 22:01:29.222977 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:29.222980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:29.223278 (+ 298us) server_negotiation.cc:407] Connection header received
0928 22:01:29.223795 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:29.223796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:29.223809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:29.223865 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:29.224450 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.225051 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.226207 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.226425 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.226512 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:29.229242 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:29.229243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:29.229244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:29.229252 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:29.231070 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:29.231643 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:29.231644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:29.231645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:29.231708 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:29.232031 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:29.232032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:29.232032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:29.232170 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:29.232219 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:29.232635 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 22:01:29.232771 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:01:29.233461 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:29.222946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46331 (local address 127.2.165.67:58593)
0928 22:01:29.223136 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:29.223178 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:29.223686 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:29.223950 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:29.223953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:29.224135 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:29.224329 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:29.224334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.225146 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.225147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:29.226112 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:29.226116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.226621 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.226622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:29.226773 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:29.227488 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:29.227500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:29.229138 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:29.231174 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:29.231181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:29.231184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:29.231551 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:29.231785 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:29.231785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:29.231786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:29.231945 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:29.232271 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:29.232277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:29.232460 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 22:01:29.233027 (+ 567us) client_negotiation.cc:240] Negotiation successful
0928 22:01:29.233204 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":122,"threads_started":1}
I20260928 22:01:29.233942 15930 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46331
I20260928 22:01:29.234058 15930 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:29.234261 15930 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
W20260928 22:01:29.234993 15930 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:46331 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:01:29.235067 15930 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:29.235133 15930 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:29.235512 15930 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:29.235586 15930 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
W20260928 22:01:29.235908 15930 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:01:29.242451 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:29.227566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.2.165.126:46331)
0928 22:01:29.227893 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:29.227895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:29.227903 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:01:29.227923 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:29.227924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:29.227934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:29.228020 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:29.228883 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.233038 (+ 4155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.236024 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:29.236366 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:29.236455 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:29.239510 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:29.239512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:29.239512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:29.239520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:29.241167 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:29.241628 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:29.241630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:29.241631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:29.241694 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:29.241929 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:29.241930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:29.241930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:29.242049 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:29.242189 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:29.242293 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 22:01:29.242374 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":59,"threads_started":1}
Sep 28 22:01:29 dist-test-slave-3t36 krb5kdc[14468](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:29.828076 15662 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:29.828240 15662 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:29.828776 15456 ts_manager.cc:194] Registered new tserver with Master: ff68b0817fb8471e854b91f5c131437e (127.2.165.65:38761)
I20260928 22:01:29.829842 15456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:43205
I20260928 22:01:30.017619 15796 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:30.017818 15796 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:30.018251 15456 ts_manager.cc:194] Registered new tserver with Master: eddfda17bb7549bf80de8451d80b5fc1 (127.2.165.66:35573)
I20260928 22:01:30.018695 15456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:60159
Sep 28 22:01:30 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:30.236795 15930 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:30.236953 15930 heartbeater.cc:511] Master 127.2.165.126:46331 requested a full tablet report, sending...
I20260928 22:01:30.238849 15456 ts_manager.cc:194] Registered new tserver with Master: 45a09043c6054eb8bfccf5edd0aa1fb6 (127.2.165.67:37637)
I20260928 22:01:30.239389 15456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:58593
I20260928 22:01:30.243969 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:01:30.253402 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.245062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.2.165.126:46331)
0928 22:01:30.245258 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:30.245260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:30.245436 (+ 176us) server_negotiation.cc:407] Connection header received
0928 22:01:30.245460 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:30.245461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:30.245473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:30.245530 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:30.246270 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.247075 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.248107 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.248425 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.248486 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.250056 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:30.250058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:30.250058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:30.250066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:30.251836 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:30.252212 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:30.252212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:30.252213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:30.252267 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:30.252476 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:30.252477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:30.252478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:30.252607 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:30.252729 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:30.253191 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 22:01:30.253295 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":62,"threads_started":1}
I20260928 22:01:30.255936 15456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:01:30.258761 15456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:01:30.276654 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.267263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:35573 (local address 127.0.0.1:38196)
0928 22:01:30.268554 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.268564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:30.268603 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:30.268884 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:30.268887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:30.268889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:30.268964 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.268966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.273089 (+ 4123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.273090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.274177 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.274179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.276328 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.276329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.276423 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.276437 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:01:30.276505 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:01:30.276558 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"thread_start_us":65,"threads_started":1}
I20260928 22:01:30.277567 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.267688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.2.165.67:37637)
0928 22:01:30.268004 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:30.268010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:30.268340 (+ 330us) server_negotiation.cc:407] Connection header received
0928 22:01:30.268378 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:30.268381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:30.268397 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:30.268461 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:30.270308 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.271371 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.274428 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.275214 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.275301 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.275370 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:30.277436 (+ 2066us) server_negotiation.cc:299] Negotiation successful
0928 22:01:30.277487 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":51,"threads_started":1}
I20260928 22:01:30.278203 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.267817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.2.165.66:35573)
0928 22:01:30.268062 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:30.268066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:30.268654 (+ 588us) server_negotiation.cc:407] Connection header received
0928 22:01:30.268717 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:30.268719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:30.268733 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:30.268800 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:30.269037 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.273011 (+ 3974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.275409 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.276159 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.276226 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.276290 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:30.278072 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0928 22:01:30.278124 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":59,"threads_started":1}
I20260928 22:01:30.277216 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.267051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:37637 (local address 127.0.0.1:49774)
0928 22:01:30.268127 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.268170 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:30.268243 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:30.270164 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:30.270165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:30.270167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:30.270211 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.270212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.271468 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.271469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.272812 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.272815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.276982 (+ 4167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.276983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.277048 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.277051 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:01:30.277105 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:01:30.277144 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":73,"threads_started":1}
I20260928 22:01:30.279305 15864 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a (DEFAULT_TABLE table=test-table [id=a2e8b4035b1c46489259f0c36a7a1e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:30.279706 15864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:30.286386 15730 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a (DEFAULT_TABLE table=test-table [id=a2e8b4035b1c46489259f0c36a7a1e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:30.286792 15730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:30.289958 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.267437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38761 (local address 127.0.0.1:59132)
0928 22:01:30.274316 (+ 6879us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.274327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:30.274364 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:30.286310 (+ 11946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:30.286312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:30.286315 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:30.286369 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.286370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.287606 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.287607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.288731 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.288735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.289717 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.289718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.289781 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.289787 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:01:30.289821 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 22:01:30.289876 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6855,"thread_start_us":59,"threads_started":1}
I20260928 22:01:30.290364 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.267553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.2.165.65:38761)
0928 22:01:30.286038 (+ 18485us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:30.286043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:30.286054 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:01:30.286089 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:30.286091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:30.286108 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:30.286191 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:30.286450 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.287500 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.288838 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.289600 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.290136 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.290200 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:30.290227 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 22:01:30.290288 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18446,"thread_start_us":58,"threads_started":1}
I20260928 22:01:30.290488 15964 tablet_bootstrap.cc:492] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: Bootstrap starting.
I20260928 22:01:30.290535 15963 tablet_bootstrap.cc:492] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: Bootstrap starting.
I20260928 22:01:30.291021 15592 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a (DEFAULT_TABLE table=test-table [id=a2e8b4035b1c46489259f0c36a7a1e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:30.291144 15964 tablet_bootstrap.cc:654] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:30.291144 15963 tablet_bootstrap.cc:654] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:30.291349 15592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:30.291467 15963 log.cc:824] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:30.291467 15964 log.cc:824] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:30.292215 15964 tablet_bootstrap.cc:492] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: No bootstrap required, opened a new log
I20260928 22:01:30.292295 15964 ts_tablet_manager.cc:1397] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:01:30.292392 15963 tablet_bootstrap.cc:492] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: No bootstrap required, opened a new log
I20260928 22:01:30.292450 15963 ts_tablet_manager.cc:1397] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:01:30.294513 15964 raft_consensus.cc:348] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.294703 15964 raft_consensus.cc:374] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:30.294739 15964 raft_consensus.cc:729] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddfda17bb7549bf80de8451d80b5fc1, State: Initialized, Role: FOLLOWER
I20260928 22:01:30.294849 15964 consensus_queue.cc:260] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.295095 15964 ts_tablet_manager.cc:1428] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:01:30.295207 15796 heartbeater.cc:503] Master 127.2.165.126:46331 was elected leader, sending a full tablet report...
I20260928 22:01:30.295037 15963 raft_consensus.cc:348] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.295403 15963 raft_consensus.cc:374] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:30.295445 15963 raft_consensus.cc:729] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a09043c6054eb8bfccf5edd0aa1fb6, State: Initialized, Role: FOLLOWER
I20260928 22:01:30.295565 15963 consensus_queue.cc:260] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.296201 15963 ts_tablet_manager.cc:1428] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 22:01:30.296429 15930 heartbeater.cc:503] Master 127.2.165.126:46331 was elected leader, sending a full tablet report...
I20260928 22:01:30.297078 15967 tablet_bootstrap.cc:492] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e: Bootstrap starting.
I20260928 22:01:30.297690 15967 tablet_bootstrap.cc:654] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:30.297991 15967 log.cc:824] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:30.298714 15967 tablet_bootstrap.cc:492] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e: No bootstrap required, opened a new log
I20260928 22:01:30.298799 15967 ts_tablet_manager.cc:1397] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:01:30.300556 15967 raft_consensus.cc:348] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.300908 15967 raft_consensus.cc:374] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:30.300997 15967 raft_consensus.cc:729] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff68b0817fb8471e854b91f5c131437e, State: Initialized, Role: FOLLOWER
I20260928 22:01:30.301174 15967 consensus_queue.cc:260] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.301501 15967 ts_tablet_manager.cc:1428] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:01:30.301642 15662 heartbeater.cc:503] Master 127.2.165.126:46331 was elected leader, sending a full tablet report...
W20260928 22:01:30.310920 15663 tablet.cc:2369] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:01:30.437867 15969 raft_consensus.cc:482] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:01:30.438036 15969 raft_consensus.cc:504] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.438488 15969 leader_election.cc:288] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff68b0817fb8471e854b91f5c131437e (127.2.165.65:38761), eddfda17bb7549bf80de8451d80b5fc1 (127.2.165.66:35573)
I20260928 22:01:30.442770 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.438993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:37537 (local address 127.2.165.66:35573)
0928 22:01:30.439044 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:30.439046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:30.439165 (+ 119us) server_negotiation.cc:407] Connection header received
0928 22:01:30.439262 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:30.439263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:30.439272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:30.439300 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:30.439533 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.440235 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.441779 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.442329 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.442398 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.442409 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:30.442648 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 22:01:30.442693 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:01:30.443125 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.438931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:33231 (local address 127.2.165.65:38761)
0928 22:01:30.438974 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:30.438977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:30.439098 (+ 121us) server_negotiation.cc:407] Connection header received
0928 22:01:30.439213 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:30.439215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:30.439227 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:30.439256 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:30.439477 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.440208 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.442177 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.442742 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.442806 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.442814 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:30.443014 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 22:01:30.443057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:01:30.443259 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.438784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:35573 (local address 127.2.165.67:37537)
0928 22:01:30.439102 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.439112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:30.439188 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:30.439380 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:30.439381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:30.439383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:30.439449 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.439451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.440354 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.440355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.441666 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.441669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.442493 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.442494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.442560 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.442564 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:01:30.443109 (+ 545us) client_negotiation.cc:240] Negotiation successful
0928 22:01:30.443159 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":81,"threads_started":1}
I20260928 22:01:30.443610 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.438692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38761 (local address 127.2.165.67:33231)
0928 22:01:30.439030 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.439041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:30.439135 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:30.439321 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:30.439322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:30.439323 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:30.439398 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.439401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.440319 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.440320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.442062 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:30.442065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.442872 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.442873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:30.442940 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.442942 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:01:30.443485 (+ 543us) client_negotiation.cc:240] Negotiation successful
0928 22:01:30.443529 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":58,"thread_start_us":112,"threads_started":1}
I20260928 22:01:30.443662 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89bffc41a64e86bfb5dea2dbb11a5a" candidate_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddfda17bb7549bf80de8451d80b5fc1" is_pre_election: true
I20260928 22:01:30.443840 15750 raft_consensus.cc:2454] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a09043c6054eb8bfccf5edd0aa1fb6 in term 0.
I20260928 22:01:30.444074 15820 leader_election.cc:302] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a09043c6054eb8bfccf5edd0aa1fb6, eddfda17bb7549bf80de8451d80b5fc1; no voters:
I20260928 22:01:30.444125 15615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89bffc41a64e86bfb5dea2dbb11a5a" candidate_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff68b0817fb8471e854b91f5c131437e" is_pre_election: true
I20260928 22:01:30.444214 15969 raft_consensus.cc:2781] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:01:30.444260 15615 raft_consensus.cc:2454] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a09043c6054eb8bfccf5edd0aa1fb6 in term 0.
I20260928 22:01:30.444286 15969 raft_consensus.cc:482] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:01:30.444310 15969 raft_consensus.cc:3037] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:30.445201 15969 raft_consensus.cc:504] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.445336 15969 leader_election.cc:288] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [CANDIDATE]: Term 1 election: Requested vote from peers ff68b0817fb8471e854b91f5c131437e (127.2.165.65:38761), eddfda17bb7549bf80de8451d80b5fc1 (127.2.165.66:35573)
I20260928 22:01:30.445509 15615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89bffc41a64e86bfb5dea2dbb11a5a" candidate_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff68b0817fb8471e854b91f5c131437e"
I20260928 22:01:30.445561 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89bffc41a64e86bfb5dea2dbb11a5a" candidate_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddfda17bb7549bf80de8451d80b5fc1"
I20260928 22:01:30.445590 15615 raft_consensus.cc:3037] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:30.445629 15750 raft_consensus.cc:3037] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:30.446417 15750 raft_consensus.cc:2454] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a09043c6054eb8bfccf5edd0aa1fb6 in term 1.
I20260928 22:01:30.446503 15615 raft_consensus.cc:2454] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a09043c6054eb8bfccf5edd0aa1fb6 in term 1.
I20260928 22:01:30.446580 15820 leader_election.cc:302] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a09043c6054eb8bfccf5edd0aa1fb6, eddfda17bb7549bf80de8451d80b5fc1; no voters:
I20260928 22:01:30.446684 15969 raft_consensus.cc:2781] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:30.446830 15969 raft_consensus.cc:686] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 1 LEADER]: Becoming Leader. State: Replica: 45a09043c6054eb8bfccf5edd0aa1fb6, State: Running, Role: LEADER
I20260928 22:01:30.446911 15969 consensus_queue.cc:237] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } }
I20260928 22:01:30.447628 15456 catalog_manager.cc:5495] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45a09043c6054eb8bfccf5edd0aa1fb6 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a09043c6054eb8bfccf5edd0aa1fb6" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 37637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 } health_report { overall_health: UNKNOWN } } }
W20260928 22:01:30.473349 15931 tablet.cc:2369] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:01:30.497555 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.494707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.2.165.67:37637)
0928 22:01:30.494807 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:30.494809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:30.494822 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:01:30.494869 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:30.494870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:30.494894 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:30.495000 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:01:30.495123 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.495613 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.496453 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:30.496707 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:30.496769 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:30.496982 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:01:30.497119 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:01:30.497202 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:30.497450 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 22:01:30.497474 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 22:01:30.500022 15615 raft_consensus.cc:1261] T 4f89bffc41a64e86bfb5dea2dbb11a5a P ff68b0817fb8471e854b91f5c131437e [term 1 FOLLOWER]: Refusing update from remote peer 45a09043c6054eb8bfccf5edd0aa1fb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:30.500022 15750 raft_consensus.cc:1261] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 1 FOLLOWER]: Refusing update from remote peer 45a09043c6054eb8bfccf5edd0aa1fb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:30.500473 15969 consensus_queue.cc:1035] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eddfda17bb7549bf80de8451d80b5fc1" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:30.500639 15969 consensus_queue.cc:1035] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff68b0817fb8471e854b91f5c131437e" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:01:30.504511 15797 tablet.cc:2369] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 22:01:30 dist-test-slave-3t36 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for HTTP/127.2.165.84@KRBTEST.COM
I20260928 22:01:30.509004 15456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:30.509110 15456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:30.510007 15456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b0c62cf757b40e8ac5524e51cd677f5: Sending DeleteTablet for 3 replicas of tablet 4f89bffc41a64e86bfb5dea2dbb11a5a
I20260928 22:01:30.510411 15592 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:30 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:59132
I20260928 22:01:30.510819 15730 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:30 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:38196
I20260928 22:01:30.511263 15864 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:30 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:49774
I20260928 22:01:30.511888 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 15502
I20260928 22:01:30.512490 15987 tablet_replica.cc:330] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: stopping tablet replica
I20260928 22:01:30.512602 15987 raft_consensus.cc:2229] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:01:30.512725 15987 raft_consensus.cc:2258] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:30.512899 15986 tablet_replica.cc:330] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1: stopping tablet replica
I20260928 22:01:30.513006 15986 raft_consensus.cc:2229] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:01:30.513075 15986 raft_consensus.cc:2258] T 4f89bffc41a64e86bfb5dea2dbb11a5a P eddfda17bb7549bf80de8451d80b5fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:30.517531 15987 ts_tablet_manager.cc:1905] T 4f89bffc41a64e86bfb5dea2dbb11a5a P 45a09043c6054eb8bfccf5edd0aa1fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:01:30.519752 15817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:38761 (error 108)
W20260928 22:01:30.524742 15448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:38761 (error 108)
I20260928 22:01:30.525519 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.525369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38761 (local address 127.0.0.1:59134)
0928 22:01:30.525439 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.525479 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:38761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:01:30.525539 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 15666
I20260928 22:01:30.529193 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.529068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:38761 (local address 127.2.165.67:42021)
0928 22:01:30.529121 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.529156 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:38761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:01:30.529459 15448 catalog_manager.cc:4584] TS ff68b0817fb8471e854b91f5c131437e (127.2.165.65:38761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f89bffc41a64e86bfb5dea2dbb11a5a: Network error: Client connection negotiation failed: client connection to 127.2.165.65:38761: connect: Connection refused (error 111)
I20260928 22:01:30.537061 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:30.536929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:35573 (local address 127.0.0.1:38198)
0928 22:01:30.536991 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:30.537019 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.66:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:01:30.538601 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 15799
I20260928 22:01:30.544304 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 15431
I20260928 22:01:30.552137 2709 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:01:30.896888 2709 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:01:30.897003 2709 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:01:31.241963 2709 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:01:31.242 UTC [14498] LOG: received smart shutdown request
2026-09-28 22:01:31.246 UTC [14498] LOG: background worker "logical replication launcher" (PID 14507) exited with exit code 1
2026-09-28 22:01:31.247 UTC [14502] LOG: shutting down
2026-09-28 22:01:31.442 UTC [14498] LOG: database system is shut down
2026-09-28T22:01:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50260 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:01:31 dist-test-slave-3t36 krb5kdc[16000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:01:31 dist-test-slave-3t36 krb5kdc[16000](info): set up 2 sockets
Sep 28 22:01:31 dist-test-slave-3t36 krb5kdc[16000](info): commencing operation
krb5kdc: starting...
W20260928 22:01:32.616994 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:01:32 dist-test-slave-3t36 krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:01:32.739583 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:34157
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:34157,127.2.165.125:46233,127.2.165.124:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:32.821782 16013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:32.822062 16013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:32.822098 16013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:32.823302 16013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:32.823379 16013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:32.823410 16013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:32.823439 16013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:32.823470 16013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:32.825057 16013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:34157,127.2.165.125:46233,127.2.165.124:41443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:34157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:32.825356 16013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:32.825592 16013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:32.830683 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:32.830745 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:32.830754 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:32.830873 16013 server_base.cc:1034] running on GCE node
I20260928 22:01:32.831248 16013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:32.831581 16013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:32.832738 16013 hybrid_clock.cc:648] HybridClock initialized: now 1790632892832732 us; error 38 us; skew 500 ppm
Sep 28 22:01:32 dist-test-slave-3t36 krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:32.835382 16013 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:32.835868 16013 webserver.cc:460] Webserver started at http://127.2.165.126:45763/ using document root <none> and password file <none>
I20260928 22:01:32.836089 16013 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:32.836154 16013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:32.836313 16013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:32.837430 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "2a24d3ccde054e918c072aa674aa8037"
format_stamp: "Formatted at 2026-09-28 22:01:32 on dist-test-slave-3t36"
server_key: "3014e4955d2f61f34c21d5329ab85fb5"
server_key_iv: "7b0b141e9a9cfdeff9d63b443fce7e55"
server_key_version: "encryptionkey@0"
I20260928 22:01:32.837791 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "2a24d3ccde054e918c072aa674aa8037"
format_stamp: "Formatted at 2026-09-28 22:01:32 on dist-test-slave-3t36"
server_key: "3014e4955d2f61f34c21d5329ab85fb5"
server_key_iv: "7b0b141e9a9cfdeff9d63b443fce7e55"
server_key_version: "encryptionkey@0"
I20260928 22:01:32.839120 16013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:01:32.839946 16028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:32.840137 16013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:32.840248 16013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "2a24d3ccde054e918c072aa674aa8037"
format_stamp: "Formatted at 2026-09-28 22:01:32 on dist-test-slave-3t36"
server_key: "3014e4955d2f61f34c21d5329ab85fb5"
server_key_iv: "7b0b141e9a9cfdeff9d63b443fce7e55"
server_key_version: "encryptionkey@0"
I20260928 22:01:32.840353 16013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:32.850950 16013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:32.853363 16013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:32.853534 16013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:32.857682 16013 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:34157
I20260928 22:01:32.857717 16080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:34157 every 8 connection(s)
I20260928 22:01:32.858070 16013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:32.858958 16081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:32.860570 16081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:01:32.864964 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16013
I20260928 22:01:32.865058 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:01:32.865267 2709 external_mini_cluster.cc:1428] Setting key 1a3ecebf77054bd9660bff18b092759f
Sep 28 22:01:32 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:32.870922 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.861410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.2.165.126:34157)
0928 22:01:32.861556 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:32.861567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:32.861663 (+ 96us) server_negotiation.cc:407] Connection header received
0928 22:01:32.862252 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:32.862254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:32.862277 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:32.862383 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:32.862957 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:32.863773 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:32.864707 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:32.865076 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:32.865206 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:32.867398 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:32.867405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:32.867406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:32.867414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:32.869202 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:32.869622 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:32.869625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:32.869626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:32.869711 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:32.869957 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:32.869957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:32.869958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:32.870190 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:32.870282 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:32.870479 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 22:01:32.870653 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260928 22:01:32.870925 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.861309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34157 (local address 127.0.0.1:35014)
0928 22:01:32.861534 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:32.861581 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:32.862084 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:32.862388 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:32.862391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:32.862683 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:32.862851 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:32.862856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:32.863884 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:32.863884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:32.864579 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:32.864581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:32.865190 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:32.865192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:32.865258 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:32.865880 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:32.865889 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:32.867298 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:32.869312 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:32.869318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:32.869321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:32.869525 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:32.869792 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:32.869793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:32.869793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:32.869883 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:32.870268 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:32.870273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:32.870347 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 22:01:32.870455 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 22:01:32.870700 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":96,"thread_start_us":102,"threads_started":1}
I20260928 22:01:32.872105 16081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.165.125@KRBTEST.COM; defaulting to no policy
I20260928 22:01:32.872527 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.872371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:34984)
0928 22:01:32.872424 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:32.872488 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:01:32.872648 16029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
I20260928 22:01:32.873090 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.872998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:34996)
0928 22:01:32.873045 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:32.873070 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:01:32.873251 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:46233: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab.
Entry for principal kudu/127.2.165.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab.
I20260928 22:01:32.892112 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:46233
--webserver_interface=127.2.165.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab
--principal=kudu/127.2.165.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:34157,127.2.165.125:46233,127.2.165.124:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:32.922485 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } attempt: 1
I20260928 22:01:32.922995 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.922855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:35010)
0928 22:01:32.922922 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:32.922964 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 22:01:32.923331 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.923237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:35014)
0928 22:01:32.923282 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:32.923308 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:01:32.923432 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:46233: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
W20260928 22:01:32.974140 16088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:32.974402 16088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:32.974452 16088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:32.975636 16088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:32.975713 16088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:32.975737 16088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:32.975759 16088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:32.975788 16088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:32.977385 16088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:34157,127.2.165.125:46233,127.2.165.124:41443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab
--principal=kudu/127.2.165.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.165.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:32.977690 16088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:32.977931 16088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:32.983383 16094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:32.983368 16093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:32.983397 16088 server_base.cc:1034] running on GCE node
W20260928 22:01:32.983368 16096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:32.983937 16088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:32.984340 16088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:32.985509 16088 hybrid_clock.cc:648] HybridClock initialized: now 1790632892985501 us; error 42 us; skew 500 ppm
Sep 28 22:01:32 dist-test-slave-3t36 krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:32.988106 16088 init.cc:377] Logged in from keytab as kudu/127.2.165.125@KRBTEST.COM (short username kudu)
I20260928 22:01:32.988648 16088 webserver.cc:460] Webserver started at http://127.2.165.125:33099/ using document root <none> and password file <none>
I20260928 22:01:32.988957 16088 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:32.989028 16088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:32.989183 16088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:32.990269 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data/instance:
uuid: "73cdb8d0fbd24db2bd462a36d52ba97e"
format_stamp: "Formatted at 2026-09-28 22:01:32 on dist-test-slave-3t36"
server_key: "b045ac4efc8bc60672c652d0849422ae"
server_key_iv: "c175be35a1265cfbfe2d0bf6f67e53bd"
server_key_version: "encryptionkey@0"
I20260928 22:01:32.990646 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/wal/instance:
uuid: "73cdb8d0fbd24db2bd462a36d52ba97e"
format_stamp: "Formatted at 2026-09-28 22:01:32 on dist-test-slave-3t36"
server_key: "b045ac4efc8bc60672c652d0849422ae"
server_key_iv: "c175be35a1265cfbfe2d0bf6f67e53bd"
server_key_version: "encryptionkey@0"
I20260928 22:01:32.991566 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } attempt: 2
I20260928 22:01:32.991956 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.991835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:35024)
0928 22:01:32.991894 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:32.991929 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:01:32.992043 16088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:01:32.992288 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:32.992191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:35038)
0928 22:01:32.992237 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:32.992262 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:01:32.992376 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:46233: Network error: Client connection negotiation failed: client connection to 127.2.165.125:46233: connect: Connection refused (error 111)
I20260928 22:01:32.992941 16103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:32.993163 16088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:32.993256 16088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/wal
uuid: "73cdb8d0fbd24db2bd462a36d52ba97e"
format_stamp: "Formatted at 2026-09-28 22:01:32 on dist-test-slave-3t36"
server_key: "b045ac4efc8bc60672c652d0849422ae"
server_key_iv: "c175be35a1265cfbfe2d0bf6f67e53bd"
server_key_version: "encryptionkey@0"
I20260928 22:01:32.993331 16088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:33.001398 16088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:33.003334 16088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:33.003500 16088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:33.007810 16088 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.125:46233
I20260928 22:01:33.007911 16155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.125:46233 every 8 connection(s)
I20260928 22:01:33.008345 16088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
I20260928 22:01:33.009440 16156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:33.011068 16156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:33.017206 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16088
I20260928 22:01:33.017292 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-1/wal/instance
I20260928 22:01:33.017477 2709 external_mini_cluster.cc:1428] Setting key 9a6f8664d6a1ec2c58ec78faaebe0884
I20260928 22:01:33.020843 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.011815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.2.165.126:34157)
0928 22:01:33.011907 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.011909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.012076 (+ 167us) server_negotiation.cc:407] Connection header received
0928 22:01:33.012547 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.012548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.012561 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.012614 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.013335 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.013841 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.014881 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.015098 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.015171 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.017600 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.017602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.017602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.017611 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.019239 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.019819 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.019820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.019821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.019878 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.020167 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.020168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.020168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.020275 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.020325 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.020654 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.020779 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 22:01:33.021133 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.011767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34157 (local address 127.0.0.1:35018)
0928 22:01:33.011955 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.011991 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.012456 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.012726 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.012730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.013045 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.013245 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.013250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.013936 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.013937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.014780 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.014782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.015199 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.015200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.015328 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.015917 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.015934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.017467 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.019382 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.019390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.019393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.019737 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.019953 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.019954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.019955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.020101 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.020340 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.020346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.020520 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.020636 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.020849 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260928 22:01:33.021795 16156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.165.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.124@KRBTEST.COM" created.
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:01:33.032212 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.022141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:35054)
0928 22:01:33.022198 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.022209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.022293 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.022509 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.022510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.022664 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.022708 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.022710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.023613 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.023614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.024115 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.024116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.024648 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.024649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.024748 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.025265 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.025269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.028876 (+ 3607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.031123 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.031124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.031125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.031293 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.031546 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.031547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.031547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.031600 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.031961 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.031962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.032000 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.032051 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.032137 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:01:33.032384 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.022200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.2.165.125:46233)
0928 22:01:33.022359 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.022364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.022387 (+ 23us) server_negotiation.cc:407] Connection header received
0928 22:01:33.022413 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.022415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.022426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.022509 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.022791 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.023500 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.024210 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.024554 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.024650 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.029011 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.029024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.029024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.029033 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.031031 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.031409 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.031413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.031414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.031479 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.031663 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.031664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.031665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.031887 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.031973 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.032087 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.032209 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260928 22:01:33.033237 16156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:01:33.033663 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.033512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53390)
0928 22:01:33.033565 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.033625 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:01:33.033857 16106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
I20260928 22:01:33.034332 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.034221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53402)
0928 22:01:33.034271 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.034304 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:01:33.034495 16156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:41443: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab.
Entry for principal kudu/127.2.165.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab.
I20260928 22:01:33.045344 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:41443
--webserver_interface=127.2.165.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab
--principal=kudu/127.2.165.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:34157,127.2.165.125:46233,127.2.165.124:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:33.083551 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } attempt: 3
I20260928 22:01:33.083770 16156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } attempt: 1
I20260928 22:01:33.084504 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.084088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53410)
0928 22:01:33.084202 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.084248 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 22:01:33.085001 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.084880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53426)
0928 22:01:33.084937 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.084971 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 22:01:33.085214 16156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:41443: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:01:33.092084 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.084117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:35058)
0928 22:01:33.084190 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.084203 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.084271 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.084515 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.084517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.084864 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.084956 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.084966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.085831 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.085832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.086418 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.086420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.086853 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.086854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.086947 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.087474 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.087487 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.088971 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.090875 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.090876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.090877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.091077 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.091327 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.091328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.091329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.091400 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.091692 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.091693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.091750 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.091841 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.091973 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 22:01:33.092084 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.084163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.2.165.125:46233)
0928 22:01:33.084206 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.084209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.084278 (+ 69us) server_negotiation.cc:407] Connection header received
0928 22:01:33.084344 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.084345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.084356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.084481 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.085065 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.085727 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.086516 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.086756 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.086854 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.089073 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.089076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.089076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.089085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.090744 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.091164 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.091166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.091166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.091233 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.091479 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.091480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.091481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.091616 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.091702 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.091885 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.091974 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:01:33.092469 16081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:01:33.092935 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.092813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53442)
0928 22:01:33.092872 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.092901 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:01:33.093320 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.093223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53454)
0928 22:01:33.093279 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.093304 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 22:01:33.093433 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:41443: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
I20260928 22:01:33.124593 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } attempt: 1
I20260928 22:01:33.125159 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.125016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53464)
0928 22:01:33.125089 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.125130 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 22:01:33.125502 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.125402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53476)
0928 22:01:33.125448 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.125473 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 22:01:33.125618 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:41443: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
W20260928 22:01:33.129870 16163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:33.130085 16163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:33.130110 16163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:33.131212 16163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:33.131248 16163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:33.131263 16163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:33.131278 16163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:33.131289 16163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:33.132809 16163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:34157,127.2.165.125:46233,127.2.165.124:41443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab
--principal=kudu/127.2.165.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.165.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:33.133024 16163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:33.133203 16163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:33.138154 16171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:33.138154 16169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:33.138164 16168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:33.138739 16163 server_base.cc:1034] running on GCE node
I20260928 22:01:33.139001 16163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:33.139288 16163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:33.140476 16163 hybrid_clock.cc:648] HybridClock initialized: now 1790632893140462 us; error 33 us; skew 500 ppm
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:33.143235 16163 init.cc:377] Logged in from keytab as kudu/127.2.165.124@KRBTEST.COM (short username kudu)
I20260928 22:01:33.143801 16163 webserver.cc:460] Webserver started at http://127.2.165.124:39091/ using document root <none> and password file <none>
I20260928 22:01:33.144053 16163 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:33.144124 16163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:33.144279 16163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:33.145449 16163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data/instance:
uuid: "03e1ea47dab54047b0bc83f6672fefea"
format_stamp: "Formatted at 2026-09-28 22:01:33 on dist-test-slave-3t36"
server_key: "544f3135b2b6c602d21036912e767c84"
server_key_iv: "ccedb2dfeb729060ac1f9b7dff652192"
server_key_version: "encryptionkey@0"
I20260928 22:01:33.145815 16163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/wal/instance:
uuid: "03e1ea47dab54047b0bc83f6672fefea"
format_stamp: "Formatted at 2026-09-28 22:01:33 on dist-test-slave-3t36"
server_key: "544f3135b2b6c602d21036912e767c84"
server_key_iv: "ccedb2dfeb729060ac1f9b7dff652192"
server_key_version: "encryptionkey@0"
I20260928 22:01:33.147197 16163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:01:33.148015 16178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:33.148206 16163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:33.148316 16163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/wal
uuid: "03e1ea47dab54047b0bc83f6672fefea"
format_stamp: "Formatted at 2026-09-28 22:01:33 on dist-test-slave-3t36"
server_key: "544f3135b2b6c602d21036912e767c84"
server_key_iv: "ccedb2dfeb729060ac1f9b7dff652192"
server_key_version: "encryptionkey@0"
I20260928 22:01:33.148397 16163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:33.153354 16156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } attempt: 2
I20260928 22:01:33.153803 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.153673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53488)
0928 22:01:33.153731 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.153772 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:01:33.154197 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.154115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53492)
0928 22:01:33.154157 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.154180 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:01:33.154366 16156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:41443: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
I20260928 22:01:33.199975 16163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:33.200757 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } attempt: 2
I20260928 22:01:33.201566 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.201383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53504)
0928 22:01:33.201482 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.201524 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 22:01:33.202008 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.201891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53508)
0928 22:01:33.201947 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.201981 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:01:33.202262 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:41443: Network error: Client connection negotiation failed: client connection to 127.2.165.124:41443: connect: Connection refused (error 111)
I20260928 22:01:33.202900 16163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:33.203076 16163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:33.207269 16163 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.124:41443
I20260928 22:01:33.207341 16230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.124:41443 every 8 connection(s)
I20260928 22:01:33.207707 16163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
I20260928 22:01:33.208640 16231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:33.210322 16231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:01:33.210813 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16163
I20260928 22:01:33.210960 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632730261496-2709-0/minicluster-data/master-2/wal/instance
I20260928 22:01:33.211184 2709 external_mini_cluster.cc:1428] Setting key 7e651b1f989cec28f83a1cbb045c56ae
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:01:33.221536 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.211111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.2.165.126:34157)
0928 22:01:33.211172 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.211174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.211428 (+ 254us) server_negotiation.cc:407] Connection header received
0928 22:01:33.212083 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.212084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.212096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.212164 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.212967 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.213749 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.214961 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.215234 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.215313 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.217896 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.217898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.217898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.217906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.219625 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.220168 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.220169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.220169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.220257 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.220625 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.220626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.220627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.220850 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.220933 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.221257 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.221420 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:01:33.221637 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.211614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.2.165.124:41443)
0928 22:01:33.211755 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.211762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.211826 (+ 64us) server_negotiation.cc:407] Connection header received
0928 22:01:33.211987 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.211993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.212013 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.212134 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.212966 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.214105 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.215096 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.215540 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.215592 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.218391 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.218397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.218398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.218403 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.220171 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.220554 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.220557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.220557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.220613 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.220877 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.220878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.220878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.221118 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.221289 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.221406 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.221520 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260928 22:01:33.221637 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.211105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:34157 (local address 127.0.0.1:35032)
0928 22:01:33.211277 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.211337 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.211989 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.212186 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.212190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.212570 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.212860 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.212867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.213844 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.213844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.214854 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.214857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.215323 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.215324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.215478 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.216045 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.216055 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.217800 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.219706 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.219711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.219713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.220082 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.220393 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.220394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.220395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.220531 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.220929 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.220935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.221109 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.221204 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.221384 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":36,"thread_start_us":79,"threads_started":1}
I20260928 22:01:33.222333 16231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:33.229597 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.222723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.2.165.125:46233)
0928 22:01:33.222764 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.222766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.222806 (+ 40us) server_negotiation.cc:407] Connection header received
0928 22:01:33.222877 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.222878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.222887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.222932 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.223192 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.223670 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.224362 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.224575 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.224658 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.226547 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.226549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.226549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.226556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.228244 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.228603 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.228604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.228604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.228655 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.228971 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.228973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.228973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.229127 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.229206 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.229417 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.229505 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:01:33.229600 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.222674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:46233 (local address 127.0.0.1:35070)
0928 22:01:33.222768 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.222778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.222819 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.222934 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.222935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.223068 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.223114 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.223116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.223750 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.223751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.224280 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.224281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.224657 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.224658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.224748 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.225216 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.225220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.226459 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.228326 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.228327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.228328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.228526 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.228801 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.228803 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.228803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.228871 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.229209 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.229210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.229279 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.229364 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.229473 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 22:01:33.230116 16231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:01:33.230316 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.222587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.2.165.126:34157)
0928 22:01:33.222648 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.222649 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.222658 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:01:33.222678 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.222679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.222690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.222746 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.223207 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.223665 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.224360 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.224562 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.224640 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.227152 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.227153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.227154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.227161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.228879 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.229260 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.229261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.229261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.229323 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.229541 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.229542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.229543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.229678 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.229918 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.230125 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.230212 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:01:33.237311 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.230418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53530)
0928 22:01:33.230524 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.230533 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.230566 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.230696 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.230696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.230848 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.230886 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.230889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.231507 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.231508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.231976 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.231977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.232347 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.232348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.232395 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.232837 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.232842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.234318 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.236024 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.236026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.236026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.236241 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.236520 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.236521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.236521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.236590 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.236952 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.236953 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.236992 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.237072 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.237172 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 22:01:33.237318 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.230497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.2.165.124:41443)
0928 22:01:33.230607 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.230608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.230616 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:01:33.230632 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.230632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.230640 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.230685 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.230972 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.231435 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.232066 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.232272 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.232341 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.234423 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.234424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.234425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.234431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.235924 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.236387 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.236388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.236389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.236443 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.236740 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.236742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.236742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.236855 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.236941 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.237107 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.237220 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":93}
I20260928 22:01:33.237999 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.230900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.2.165.125:46233)
0928 22:01:33.230941 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.230943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.230953 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:01:33.230985 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.230986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.230994 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.231037 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.231507 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.231950 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.232605 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.232867 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.232957 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.235209 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.235210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.235211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.235218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.236819 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.237158 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.237159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.237160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.237211 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.237449 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.237450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.237451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.237599 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.237783 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.237824 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.237920 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:01:33.239161 16231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea: Bootstrap starting.
I20260928 22:01:33.239703 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:01:33.240134 16231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:33.240341 16231 log.cc:824] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:33.241325 16231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea: No bootstrap required, opened a new log
I20260928 22:01:33.243480 16231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.243628 16231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:33.243659 16231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e1ea47dab54047b0bc83f6672fefea, State: Initialized, Role: FOLLOWER
I20260928 22:01:33.243788 16231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.244103 16231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:33.245580 16156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } attempt: 3
W20260928 22:01:33.246515 16255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:01:33.246647 16255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:01:33.246860 16244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.247077 16244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:01:33.248824 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.241021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.2.165.126:34157)
0928 22:01:33.241072 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.241074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.241191 (+ 117us) server_negotiation.cc:407] Connection header received
0928 22:01:33.241314 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.241315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.241329 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.241361 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.241955 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.242792 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.243760 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.244082 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.244145 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.245831 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.245832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.245833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.245840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.247558 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.247927 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.247928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.247929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.247981 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.248202 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.248202 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.248203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.248323 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.248407 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.248618 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.248728 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:01:33.246953 16163 master_runner.cc:186] Error getting master registration for 127.2.165.126:34157: OK, instance_id { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" instance_seqno: 1790632892856391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:01:33.249877 16163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:01:33.252769 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.241300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.2.165.124:41443)
0928 22:01:33.241336 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.241338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.242053 (+ 715us) server_negotiation.cc:407] Connection header received
0928 22:01:33.242138 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.242138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.242148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.242174 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.243035 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.243510 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.244516 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.244841 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.244887 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.249495 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.249497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.249497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.249504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.251362 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.251856 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.251857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.251857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.251917 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.252127 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.252128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.252128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.252260 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.252365 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.252549 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.252627 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:01:33.255061 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.241017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.2.165.125:46233)
0928 22:01:33.241053 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.241055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.241314 (+ 259us) server_negotiation.cc:407] Connection header received
0928 22:01:33.241408 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.241409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.241419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.241465 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.242112 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.242783 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.245851 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.246128 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.246174 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.251398 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.251400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.251401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.251421 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.253649 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.254025 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.254026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.254027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.254106 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.254526 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.254527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.254528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.254636 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.254775 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.254884 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.254966 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:01:33.257009 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.245815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53546)
0928 22:01:33.245896 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.245906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.245956 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.246600 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.246602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.247042 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.247086 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.247088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.248136 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.248137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.248929 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.248932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.250172 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.250174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.250699 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.251099 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.251105 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.252957 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.256023 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.256024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.256024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.256183 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.256412 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.256413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.256413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.256471 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.256769 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.256769 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.256813 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.256863 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.256947 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 22:01:33.257292 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.246027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.2.165.124:41443)
0928 22:01:33.246456 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.246458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.246467 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:01:33.246487 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.246488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.246498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.247166 (+ 668us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.247235 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.248045 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.249023 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.250073 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.251876 (+ 1803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.254046 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.254048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.254048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.254055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.255934 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.256256 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.256257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.256258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.256332 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.256536 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.256537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.256537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.256710 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.256779 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.256913 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.256997 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":366,"threads_started":1}
I20260928 22:01:33.258906 16156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: Bootstrap starting.
I20260928 22:01:33.259583 16156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:33.259886 16156 log.cc:824] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:33.260856 16156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: No bootstrap required, opened a new log
I20260928 22:01:33.262452 16156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.262566 16156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:33.262588 16156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cdb8d0fbd24db2bd462a36d52ba97e, State: Initialized, Role: FOLLOWER
I20260928 22:01:33.262725 16156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.263016 16259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.263067 16156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:33.263121 16259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:01:33.264884 16270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:01:33.265012 16270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:01:33.265091 16088 master_runner.cc:186] Error getting master registration for 127.2.165.126:34157: OK, instance_id { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" instance_seqno: 1790632892856391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:01:33.268098 16088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:01:33.301496 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } attempt: 3
Sep 28 22:01:33 dist-test-slave-3t36 krb5kdc[16000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:01:33.309942 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.301821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:41443 (local address 127.0.0.1:53554)
0928 22:01:33.301875 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:33.301886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:33.301934 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:33.302100 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:33.302101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:33.302298 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:33.302377 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.302379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.303382 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.303383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.304197 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:33.304199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.304725 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.304726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:33.304789 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.305334 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:33.305339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:33.306894 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:33.308817 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.308819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.308819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.309048 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.309244 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:33.309246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:33.309246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:33.309325 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:33.309611 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:33.309612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:33.309663 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 22:01:33.309734 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:01:33.309836 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:01:33.309973 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:33.301886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.2.165.124:41443)
0928 22:01:33.301931 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:33.301933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:33.301943 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:01:33.302000 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:33.302001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:33.302011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:33.302080 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:33.302471 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.303289 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.304297 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:33.304624 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:33.304720 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:33.306990 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:33.306992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:33.306993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:33.307001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:33.308719 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.309128 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.309129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.309130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.309184 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:33.309396 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:33.309397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:33.309397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:33.309538 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:33.309610 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:33.309776 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 22:01:33.309869 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:01:33.311873 16081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: Bootstrap starting.
I20260928 22:01:33.312637 16081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:33.313000 16081 log.cc:824] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:33.313896 16081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: No bootstrap required, opened a new log
I20260928 22:01:33.315973 16081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.316118 16081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:33.316150 16081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a24d3ccde054e918c072aa674aa8037, State: Initialized, Role: FOLLOWER
I20260928 22:01:33.316298 16081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.316574 16273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.316617 16081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:33.316758 16273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:01:33.318609 16284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:01:33.318727 16284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:01:33.321475 16013 master_runner.cc:263] No leader master found from master 2a24d3ccde054e918c072aa674aa8037
I20260928 22:01:33.321535 16013 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 22:01:33.354387 16034 tablet.cc:2369] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:01:33.413985 16244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:01:33.414142 16244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.414434 16244 leader_election.cc:288] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a24d3ccde054e918c072aa674aa8037 (127.2.165.126:34157), 73cdb8d0fbd24db2bd462a36d52ba97e (127.2.165.125:46233)
I20260928 22:01:33.414798 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03e1ea47dab54047b0bc83f6672fefea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" is_pre_election: true
I20260928 22:01:33.414803 16055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03e1ea47dab54047b0bc83f6672fefea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a24d3ccde054e918c072aa674aa8037" is_pre_election: true
I20260928 22:01:33.414950 16130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e1ea47dab54047b0bc83f6672fefea in term 0.
I20260928 22:01:33.414955 16055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e1ea47dab54047b0bc83f6672fefea in term 0.
I20260928 22:01:33.415138 16179 leader_election.cc:302] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e1ea47dab54047b0bc83f6672fefea, 73cdb8d0fbd24db2bd462a36d52ba97e; no voters:
I20260928 22:01:33.415277 16244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:01:33.415328 16244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:01:33.415351 16244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:33.416118 16244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.416270 16244 leader_election.cc:288] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [CANDIDATE]: Term 1 election: Requested vote from peers 2a24d3ccde054e918c072aa674aa8037 (127.2.165.126:34157), 73cdb8d0fbd24db2bd462a36d52ba97e (127.2.165.125:46233)
I20260928 22:01:33.416468 16055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03e1ea47dab54047b0bc83f6672fefea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a24d3ccde054e918c072aa674aa8037"
I20260928 22:01:33.416465 16130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03e1ea47dab54047b0bc83f6672fefea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e"
I20260928 22:01:33.416545 16055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:33.416559 16130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:33.417438 16055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e1ea47dab54047b0bc83f6672fefea in term 1.
I20260928 22:01:33.417438 16130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e1ea47dab54047b0bc83f6672fefea in term 1.
I20260928 22:01:33.417619 16179 leader_election.cc:302] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e1ea47dab54047b0bc83f6672fefea, 73cdb8d0fbd24db2bd462a36d52ba97e; no voters:
I20260928 22:01:33.417695 16244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:33.417838 16244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [term 1 LEADER]: Becoming Leader. State: Replica: 03e1ea47dab54047b0bc83f6672fefea, State: Running, Role: LEADER
I20260928 22:01:33.417937 16244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } }
I20260928 22:01:33.418339 16244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e1ea47dab54047b0bc83f6672fefea. Latest consensus state: current_term: 1 leader_uuid: "03e1ea47dab54047b0bc83f6672fefea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.418424 16244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:33.418661 16288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:33.418857 16288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:33.420341 16130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [term 1 FOLLOWER]: Refusing update from remote peer 03e1ea47dab54047b0bc83f6672fefea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:33.420358 16055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [term 1 FOLLOWER]: Refusing update from remote peer 03e1ea47dab54047b0bc83f6672fefea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:33.420702 16244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:33.420825 16286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:33.422024 16273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e1ea47dab54047b0bc83f6672fefea. Latest consensus state: current_term: 1 leader_uuid: "03e1ea47dab54047b0bc83f6672fefea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.422132 16273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:01:33.422199 16259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e1ea47dab54047b0bc83f6672fefea. Latest consensus state: current_term: 1 leader_uuid: "03e1ea47dab54047b0bc83f6672fefea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.422294 16259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:01:33.423221 16244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03e1ea47dab54047b0bc83f6672fefea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.423305 16244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:33.423394 16273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03e1ea47dab54047b0bc83f6672fefea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.423473 16273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:01:33.423552 16290 mvcc.cc:205] Tried to move back new op lower bound from 7334432331447422976 to 7334432331440517120. Current Snapshot: MvccSnapshot[applied={T|T < 7334432331447422976}]
I20260928 22:01:33.424036 16286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03e1ea47dab54047b0bc83f6672fefea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.424119 16286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:33.424140 16288 catalog_manager.cc:1347] Generated new cluster ID: 1133231aa9ae4ad98223b6a076024f5b
I20260928 22:01:33.424173 16288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:01:33.424431 16259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03e1ea47dab54047b0bc83f6672fefea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a24d3ccde054e918c072aa674aa8037" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 34157 } } peers { permanent_uuid: "73cdb8d0fbd24db2bd462a36d52ba97e" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 46233 } } peers { permanent_uuid: "03e1ea47dab54047b0bc83f6672fefea" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 41443 } } }
I20260928 22:01:33.424520 16259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:01:33.428777 16291 mvcc.cc:205] Tried to move back new op lower bound from 7334432331447422976 to 7334432331440517120. Current Snapshot: MvccSnapshot[applied={T|T < 7334432331447422976 or (T in {7334432331447422976})}]
I20260928 22:01:33.431823 16288 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:01:33.432507 16288 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:01:33.439203 16288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e1ea47dab54047b0bc83f6672fefea: Generated new TSK 0
I20260928 22:01:33.439436 16288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:01:34.265583 16270 catalog_manager.cc:1259] Loaded cluster ID: 1133231aa9ae4ad98223b6a076024f5b
I20260928 22:01:34.265748 16270 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: loading cluster ID for follower catalog manager: success
I20260928 22:01:34.266993 16270 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: acquiring CA information for follower catalog manager: success
I20260928 22:01:34.267366 16270 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73cdb8d0fbd24db2bd462a36d52ba97e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:01:34.319259 16284 catalog_manager.cc:1259] Loaded cluster ID: 1133231aa9ae4ad98223b6a076024f5b
I20260928 22:01:34.319388 16284 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: loading cluster ID for follower catalog manager: success
I20260928 22:01:34.320225 16284 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: acquiring CA information for follower catalog manager: success
I20260928 22:01:34.320506 16284 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a24d3ccde054e918c072aa674aa8037: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:01:34.506225 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16013
I20260928 22:01:34.511863 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16088
I20260928 22:01:34.517037 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16163
2026-09-28T22:01:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2964 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:01:34 dist-test-slave-3t36 krb5kdc[16304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:01:34 dist-test-slave-3t36 krb5kdc[16304](info): set up 2 sockets
Sep 28 22:01:34 dist-test-slave-3t36 krb5kdc[16304](info): commencing operation
krb5kdc: starting...
W20260928 22:01:35.560900 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:01:35 dist-test-slave-3t36 krb5kdc[16304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:01:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:01:35.689088 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:36219
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:35.772776 16317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:35.773043 16317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:35.773083 16317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:35.774294 16317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:35.774367 16317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:35.774395 16317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:35.774437 16317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:35.774468 16317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:35.776026 16317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:35.776309 16317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:35.776600 16317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:35.781904 16323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:35.781881 16325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:35.781991 16322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:35.782191 16317 server_base.cc:1034] running on GCE node
I20260928 22:01:35.782522 16317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:35.782840 16317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:35.784024 16317 hybrid_clock.cc:648] HybridClock initialized: now 1790632895784023 us; error 59 us; skew 500 ppm
Sep 28 22:01:35 dist-test-slave-3t36 krb5kdc[16304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:35.786796 16317 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:35.787258 16317 webserver.cc:460] Webserver started at http://127.2.165.126:36819/ using document root <none> and password file <none>
I20260928 22:01:35.787554 16317 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:35.787629 16317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:35.787779 16317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:35.788959 16317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "25d0f23a44244ec9b5e7b14047a5d2bc"
format_stamp: "Formatted at 2026-09-28 22:01:35 on dist-test-slave-3t36"
server_key: "8aa113b9681eedd51ec3de8267c738d3"
server_key_iv: "1b5df99b67e2b1ced262e588479d9b36"
server_key_version: "encryptionkey@0"
I20260928 22:01:35.789345 16317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "25d0f23a44244ec9b5e7b14047a5d2bc"
format_stamp: "Formatted at 2026-09-28 22:01:35 on dist-test-slave-3t36"
server_key: "8aa113b9681eedd51ec3de8267c738d3"
server_key_iv: "1b5df99b67e2b1ced262e588479d9b36"
server_key_version: "encryptionkey@0"
I20260928 22:01:35.790803 16317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:35.791666 16332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:35.791868 16317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:35.791988 16317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "25d0f23a44244ec9b5e7b14047a5d2bc"
format_stamp: "Formatted at 2026-09-28 22:01:35 on dist-test-slave-3t36"
server_key: "8aa113b9681eedd51ec3de8267c738d3"
server_key_iv: "1b5df99b67e2b1ced262e588479d9b36"
server_key_version: "encryptionkey@0"
I20260928 22:01:35.792068 16317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:35.801180 16317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:35.803840 16317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:35.804036 16317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:35.808161 16317 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:36219
I20260928 22:01:35.808246 16384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:36219 every 8 connection(s)
I20260928 22:01:35.808589 16317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:35.809500 16385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:35.811800 16385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc: Bootstrap starting.
I20260928 22:01:35.812392 16385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:35.812779 16385 log.cc:824] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:35.813550 16385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc: No bootstrap required, opened a new log
I20260928 22:01:35.814428 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16317
I20260928 22:01:35.814570 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:01:35.814752 2709 external_mini_cluster.cc:1428] Setting key a08b39934234c7ff34e9f4a84ded12f9
I20260928 22:01:35.815661 16385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36219 } }
I20260928 22:01:35.815835 16385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:35.815876 16385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d0f23a44244ec9b5e7b14047a5d2bc, State: Initialized, Role: FOLLOWER
I20260928 22:01:35.816028 16385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36219 } }
I20260928 22:01:35.816118 16385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:01:35.816141 16385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:01:35.816213 16385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:35.817369 16385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36219 } }
I20260928 22:01:35.817487 16385 leader_election.cc:302] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d0f23a44244ec9b5e7b14047a5d2bc; no voters:
I20260928 22:01:35.817634 16385 leader_election.cc:288] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:01:35.817705 16390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:35.817855 16390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [term 1 LEADER]: Becoming Leader. State: Replica: 25d0f23a44244ec9b5e7b14047a5d2bc, State: Running, Role: LEADER
I20260928 22:01:35.817868 16385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:35.817988 16390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36219 } }
I20260928 22:01:35.818552 16391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d0f23a44244ec9b5e7b14047a5d2bc. Latest consensus state: current_term: 1 leader_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36219 } } }
I20260928 22:01:35.818511 16392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d0f23a44244ec9b5e7b14047a5d2bc" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 36219 } } }
I20260928 22:01:35.818621 16391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:35.818658 16392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:35.818866 16399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:35.819417 16399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:35.821056 16399 catalog_manager.cc:1347] Generated new cluster ID: b3cae99deeb848939db14b4039e889dd
I20260928 22:01:35.821110 16399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:01:35 dist-test-slave-3t36 krb5kdc[16304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:35.826946 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:35.815308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.2.165.126:36219)
0928 22:01:35.815530 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:35.815540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:35.815583 (+ 43us) server_negotiation.cc:407] Connection header received
0928 22:01:35.815704 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:35.815711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:35.815767 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:35.815916 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:35.816658 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:35.819242 (+ 2584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:35.820095 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:35.820801 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:35.820914 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:35.823320 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:35.823324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:35.823325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:35.823337 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:35.825117 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:35.825497 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:35.825500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:35.825501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:35.825898 (+ 397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:35.826122 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:35.826123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:35.826123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:35.826359 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:35.826550 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:35.826618 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 22:01:35.826762 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260928 22:01:35.827843 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:01:35.854369 16399 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:01:35.855020 16399 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:01:35.858708 16399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d0f23a44244ec9b5e7b14047a5d2bc: Generated new TSK 0
I20260928 22:01:35.858920 16399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:01:35.882097 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16317
2026-09-28T22:01:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1363 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:01:35 dist-test-slave-3t36 krb5kdc[16412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:01:35 dist-test-slave-3t36 krb5kdc[16412](info): set up 2 sockets
Sep 28 22:01:35 dist-test-slave-3t36 krb5kdc[16412](info): commencing operation
krb5kdc: starting...
W20260928 22:01:36.940910 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:01:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:01:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:01:37.068221 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45199
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:37.151876 16425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:37.152122 16425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:37.152148 16425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:37.153425 16425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:37.153471 16425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:37.153486 16425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:37.153499 16425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:37.153512 16425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:37.154959 16425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:37.155174 16425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:37.155357 16425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:01:37.160544 16425 server_base.cc:1034] running on GCE node
W20260928 22:01:37.160511 16431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:37.160530 16433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:37.160511 16430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:37.161015 16425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:37.161540 16425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:37.162735 16425 hybrid_clock.cc:648] HybridClock initialized: now 1790632897162684 us; error 48 us; skew 500 ppm
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:37.172168 16425 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:37.172746 16425 webserver.cc:460] Webserver started at http://127.2.165.126:45215/ using document root <none> and password file <none>
I20260928 22:01:37.173010 16425 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:37.173060 16425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:37.173154 16425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:37.174283 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "e4232c423c66416fbefd46de108e2256"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "fe905cb027944eb1499a5b27ff101489"
server_key_iv: "11d622a2fbb741daa22af7d1e89c61de"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.174609 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "e4232c423c66416fbefd46de108e2256"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "fe905cb027944eb1499a5b27ff101489"
server_key_iv: "11d622a2fbb741daa22af7d1e89c61de"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.175983 16425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:01:37.176846 16440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.177028 16425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:37.177134 16425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "e4232c423c66416fbefd46de108e2256"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "fe905cb027944eb1499a5b27ff101489"
server_key_iv: "11d622a2fbb741daa22af7d1e89c61de"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.177196 16425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:37.215998 16425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:37.218535 16425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:37.218667 16425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:37.222735 16425 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45199
I20260928 22:01:37.222838 16492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45199 every 8 connection(s)
I20260928 22:01:37.223177 16425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:37.223820 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16425
I20260928 22:01:37.223968 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:01:37.224207 2709 external_mini_cluster.cc:1428] Setting key d4ba769a0dbe649b63b0710dd53a3ea3
I20260928 22:01:37.224526 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:37.227588 16493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Bootstrap starting.
I20260928 22:01:37.228214 16493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:37.228533 16493 log.cc:824] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:37.229288 16493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: No bootstrap required, opened a new log
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:37.231398 16493 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:37.231564 16493 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:37.231611 16493 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4232c423c66416fbefd46de108e2256, State: Initialized, Role: FOLLOWER
I20260928 22:01:37.231789 16493 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:37.231918 16493 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:01:37.231953 16493 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:01:37.232003 16493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:37.232725 16493 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:37.232833 16493 leader_election.cc:302] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4232c423c66416fbefd46de108e2256; no voters:
I20260928 22:01:37.233042 16493 leader_election.cc:288] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:01:37.233074 16498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:37.233182 16498 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 LEADER]: Becoming Leader. State: Replica: e4232c423c66416fbefd46de108e2256, State: Running, Role: LEADER
I20260928 22:01:37.233265 16498 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:37.233304 16493 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:37.233616 16498 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4232c423c66416fbefd46de108e2256. Latest consensus state: current_term: 1 leader_uuid: "e4232c423c66416fbefd46de108e2256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } } }
I20260928 22:01:37.233733 16498 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:37.233623 16500 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4232c423c66416fbefd46de108e2256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } } }
I20260928 22:01:37.233899 16500 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:37.234156 16504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:37.234754 16504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:37.235759 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.224839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.2.165.126:45199)
0928 22:01:37.225039 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.225048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.225088 (+ 40us) server_negotiation.cc:407] Connection header received
0928 22:01:37.225715 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.225723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.225772 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.225924 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:37.226575 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.227451 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.228305 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.228803 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.228925 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.231590 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:37.231597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:37.231598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:37.231611 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:37.233756 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.234232 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.234236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.234237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.234358 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.234612 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.234613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.234613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.234934 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:37.235049 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.235395 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.235553 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260928 22:01:37.236527 16504 catalog_manager.cc:1347] Generated new cluster ID: 07a470be8316412aad7df2d864a36f60
I20260928 22:01:37.236585 16504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:01:37.243680 16504 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:01:37.244498 16504 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:01:37.251530 16504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Generated new TSK 0
I20260928 22:01:37.251755 16504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 22:01:37.283334 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45199
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:37.370313 16519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:37.370538 16519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:37.370564 16519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:37.371801 16519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:37.371855 16519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:37.371884 16519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:01:37.373497 16519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:37.373742 16519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:37.374015 16519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:37.379196 16524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:37.379227 16525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:37.379252 16519 server_base.cc:1034] running on GCE node
W20260928 22:01:37.379227 16527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:37.379714 16519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:37.379987 16519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:37.381165 16519 hybrid_clock.cc:648] HybridClock initialized: now 1790632897381159 us; error 41 us; skew 500 ppm
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:37.383940 16519 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:01:37.384362 16519 webserver.cc:460] Webserver started at http://127.2.165.65:45027/ using document root <none> and password file <none>
I20260928 22:01:37.384614 16519 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:37.384699 16519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:37.384864 16519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:37.385932 16519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "8a540c44e67046878924c4905e0bee01"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "c36490de2939ef56dac37ad0f16b8306"
server_key_iv: "3571615a87ebe89fd49b33714be833d6"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.386317 16519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "8a540c44e67046878924c4905e0bee01"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "c36490de2939ef56dac37ad0f16b8306"
server_key_iv: "3571615a87ebe89fd49b33714be833d6"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.387768 16519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:01:37.388720 16534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.388918 16519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:37.389022 16519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "8a540c44e67046878924c4905e0bee01"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "c36490de2939ef56dac37ad0f16b8306"
server_key_iv: "3571615a87ebe89fd49b33714be833d6"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.389112 16519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:37.406143 16519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:37.409161 16519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:37.409283 16519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:37.409493 16519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:37.409833 16519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:37.409870 16519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.409893 16519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:37.409906 16519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.415848 16519 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:39557
I20260928 22:01:37.415952 16647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:39557 every 8 connection(s)
I20260928 22:01:37.416271 16519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:01:37.418665 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16519
I20260928 22:01:37.418763 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:01:37.419006 2709 external_mini_cluster.cc:1428] Setting key e94ebaf40313c57cf0e950fadb41a92c
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:37.427417 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.417062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:56397 (local address 127.2.165.126:45199)
0928 22:01:37.417128 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.417130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.417361 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:01:37.417896 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.417897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.417907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.417959 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:37.418969 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.419698 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.420716 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.421049 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.421109 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.423772 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:37.423774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:37.423775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:37.423783 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:37.425747 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.426207 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.426209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.426209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.426276 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.426601 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.426602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.426603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.426756 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:37.426831 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.427177 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.427284 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:01:37.427507 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.417051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:56397)
0928 22:01:37.417249 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.417287 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.417791 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.418014 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.418018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.418595 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:37.418886 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.418891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.419785 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.419785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.420591 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.420593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.421136 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.421137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.421294 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.421892 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:37.421902 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:37.423672 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:37.425850 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:37.425858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:37.425860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:37.426123 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:37.426390 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:37.426391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:37.426392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:37.426524 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:37.426848 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:37.426855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:37.426992 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.427122 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.427285 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
I20260928 22:01:37.428115 16648 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45199
I20260928 22:01:37.428238 16648 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:37.428437 16648 heartbeater.cc:511] Master 127.2.165.126:45199 requested a full tablet report, sending...
I20260928 22:01:37.429118 16457 ts_manager.cc:194] Registered new tserver with Master: 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557)
I20260928 22:01:37.429750 16457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:56397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 22:01:37.448333 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45199
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:37.534658 16653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:37.534951 16653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:37.534986 16653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:37.536168 16653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:37.536216 16653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:37.536242 16653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:01:37.537746 16653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:37.538002 16653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:37.538246 16653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:37.543466 16661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:37.543488 16659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:37.543596 16653 server_base.cc:1034] running on GCE node
W20260928 22:01:37.543547 16658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:37.543867 16653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:37.544152 16653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:37.545312 16653 hybrid_clock.cc:648] HybridClock initialized: now 1790632897545295 us; error 35 us; skew 500 ppm
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:37.547886 16653 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:01:37.548370 16653 webserver.cc:460] Webserver started at http://127.2.165.66:42989/ using document root <none> and password file <none>
I20260928 22:01:37.548728 16653 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:37.548811 16653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:37.548979 16653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:37.550055 16653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "abab70c4ab13484a8b7380c1965fdccf"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "2edfb713a9b5a9e255ad1ebee87749ae"
server_key_iv: "c41c68f697b9d44f943e3e122322cd58"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.550443 16653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "abab70c4ab13484a8b7380c1965fdccf"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "2edfb713a9b5a9e255ad1ebee87749ae"
server_key_iv: "c41c68f697b9d44f943e3e122322cd58"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.551877 16653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:01:37.552815 16668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.553006 16653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:37.553113 16653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "abab70c4ab13484a8b7380c1965fdccf"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "2edfb713a9b5a9e255ad1ebee87749ae"
server_key_iv: "c41c68f697b9d44f943e3e122322cd58"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.553205 16653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:37.591948 16653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:37.594280 16653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:37.594465 16653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:37.594725 16653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:37.595063 16653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:37.595122 16653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.595167 16653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:37.595192 16653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.601394 16653 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:42615
I20260928 22:01:37.601418 16781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:42615 every 8 connection(s)
I20260928 22:01:37.601841 16653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:01:37.604014 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16653
I20260928 22:01:37.604108 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:01:37.604298 2709 external_mini_cluster.cc:1428] Setting key 04f59d39839f83c87f873494c25d6384
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:37.613205 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.602675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:41621 (local address 127.2.165.126:45199)
0928 22:01:37.602739 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.602741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.602981 (+ 240us) server_negotiation.cc:407] Connection header received
0928 22:01:37.603518 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.603519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.603528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.603594 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:37.604764 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.605476 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.606738 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.606936 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.607039 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.609609 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:37.609612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:37.609612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:37.609621 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:37.611371 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.611908 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.611909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.611910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.611979 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.612309 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.612311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.612311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.612484 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:37.612570 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.612989 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.613102 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 22:01:37.613343 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.602657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:41621)
0928 22:01:37.602836 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.602880 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.603431 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.603656 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.603659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.604384 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:37.604652 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.604659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.605576 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.605576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.606632 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.606636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.607032 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.607033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.607189 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.607865 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:37.607874 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:37.609497 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:37.611534 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:37.611541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:37.611544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:37.611815 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:37.612083 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:37.612084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:37.612085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:37.612224 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:37.612579 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:37.612589 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:37.612826 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.612942 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.613101 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260928 22:01:37.614147 16782 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45199
I20260928 22:01:37.614276 16782 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:37.614583 16782 heartbeater.cc:511] Master 127.2.165.126:45199 requested a full tablet report, sending...
I20260928 22:01:37.615072 16457 ts_manager.cc:194] Registered new tserver with Master: abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615)
I20260928 22:01:37.615533 16457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:41621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 22:01:37.632949 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45199
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:37.718915 16787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:37.719141 16787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:37.719165 16787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:37.720464 16787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:37.720511 16787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:37.720541 16787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:01:37.722162 16787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:37.722378 16787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:37.722563 16787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:37.727977 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:37.727912 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:37.728057 16787 server_base.cc:1034] running on GCE node
W20260928 22:01:37.727999 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:37.728447 16787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:37.728837 16787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:37.730034 16787 hybrid_clock.cc:648] HybridClock initialized: now 1790632897730006 us; error 46 us; skew 500 ppm
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:37.732970 16787 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:01:37.733500 16787 webserver.cc:460] Webserver started at http://127.2.165.67:35707/ using document root <none> and password file <none>
I20260928 22:01:37.733762 16787 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:37.733839 16787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:37.734030 16787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:37.735385 16787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "11caba988dd046a59eed75129e684717"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "9549a4dbe2d267127475e5b5fe84e351"
server_key_iv: "80c69cb0852db7b4db548f22aaaeb962"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.735737 16787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "11caba988dd046a59eed75129e684717"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "9549a4dbe2d267127475e5b5fe84e351"
server_key_iv: "80c69cb0852db7b4db548f22aaaeb962"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.737279 16787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 22:01:37.738269 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.738517 16787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:37.738597 16787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "11caba988dd046a59eed75129e684717"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "9549a4dbe2d267127475e5b5fe84e351"
server_key_iv: "80c69cb0852db7b4db548f22aaaeb962"
server_key_version: "encryptionkey@0"
I20260928 22:01:37.738657 16787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:37.746878 16787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:37.749404 16787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:37.749537 16787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:37.749742 16787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:37.750094 16787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:37.750131 16787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.750157 16787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:37.750209 16787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:37.756942 16787 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:41247
I20260928 22:01:37.756999 16915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:41247 every 8 connection(s)
I20260928 22:01:37.757503 16787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:01:37.758594 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16787
I20260928 22:01:37.758728 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:01:37.758898 2709 external_mini_cluster.cc:1428] Setting key bf638ef1c8f84d385e5fcf9fd4aec97b
Sep 28 22:01:37 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:37.768852 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.758298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:48187 (local address 127.2.165.126:45199)
0928 22:01:37.758395 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.758397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.758559 (+ 162us) server_negotiation.cc:407] Connection header received
0928 22:01:37.759278 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.759279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.759306 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.759369 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:37.760778 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.761319 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.762495 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.762708 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.762776 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.765475 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:37.765477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:37.765477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:37.765485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:37.767120 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.767568 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.767569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.767569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.767645 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.767980 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.767982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.767982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.768117 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:37.768191 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.768638 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.768785 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 22:01:37.769001 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.758197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:48187)
0928 22:01:37.758420 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.758471 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.759149 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.759487 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.759493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.760285 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:37.760602 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.760608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.761536 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.761537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.762380 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.762382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.762810 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.762811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.762951 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.763583 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:37.763592 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:37.765325 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:37.767216 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:37.767224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:37.767226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:37.767478 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:37.767741 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:37.767742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:37.767742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:37.767868 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:37.768236 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:37.768242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:37.768379 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.768490 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.768632 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260928 22:01:37.769630 16916 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45199
I20260928 22:01:37.769719 16916 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:37.769917 16916 heartbeater.cc:511] Master 127.2.165.126:45199 requested a full tablet report, sending...
I20260928 22:01:37.770392 16457 ts_manager.cc:194] Registered new tserver with Master: 11caba988dd046a59eed75129e684717 (127.2.165.67:41247)
I20260928 22:01:37.770951 16457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:48187
I20260928 22:01:37.771052 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:01:37.779048 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.772406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.2.165.126:45199)
0928 22:01:37.772446 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.772449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.772576 (+ 127us) server_negotiation.cc:407] Connection header received
0928 22:01:37.772615 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.772616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.772624 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.772730 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:37.773379 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.773926 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.774732 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.774918 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.774985 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.776398 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:37.776399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:37.776399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:37.776406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:37.777905 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.778270 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.778270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.778271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.778333 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:37.778509 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:37.778510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:37.778510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:37.778691 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:37.778768 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.778914 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.778976 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:01:37.781105 16457 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:01:37.782787 16457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:01:37.789587 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.785451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.2.165.66:42615)
0928 22:01:37.785629 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.785635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.785653 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:01:37.785796 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.785799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.785817 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.785916 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:37.786197 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.787121 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.788359 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.789178 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.789248 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.789311 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.789461 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.789504 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260928 22:01:37.789601 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.785342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42615 (local address 127.0.0.1:60104)
0928 22:01:37.785565 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.785607 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.785673 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.786029 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.786032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.786035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:37.786112 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.786114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.787228 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.787229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.788263 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.788266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.789262 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.789263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.789340 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.789354 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.789437 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.789499 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":139,"threads_started":1}
I20260928 22:01:37.789935 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.785647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:41247 (local address 127.0.0.1:53230)
0928 22:01:37.786262 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.786272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.786321 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.786529 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.786530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.786532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:37.786591 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.786593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.787870 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.787871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.788810 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.788812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.789690 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.789691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.789771 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.789775 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.789832 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.789874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":79,"threads_started":1}
I20260928 22:01:37.789980 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.785847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.2.165.67:41247)
0928 22:01:37.786252 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.786257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.786311 (+ 54us) server_negotiation.cc:407] Connection header received
0928 22:01:37.786388 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.786390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.786424 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.786499 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:37.786673 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.787754 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.788893 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.789538 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.789601 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.789654 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.789860 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.789903 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":80,"threads_started":1}
I20260928 22:01:37.790257 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.785345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39557 (local address 127.0.0.1:44816)
0928 22:01:37.785840 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.785852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.785899 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.786443 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.786444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.786447 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:37.786502 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.786504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.787949 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.787950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.789068 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.789071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.790030 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.790031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.790096 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.790100 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.790131 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.790166 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":154,"threads_started":1}
I20260928 22:01:37.790570 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.785656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.2.165.65:39557)
0928 22:01:37.786066 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.786071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.786078 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:01:37.786112 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.786114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.786126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.786424 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:37.786596 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.787756 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.789178 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.789927 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.790359 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.790427 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.790452 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.790495 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":93,"threads_started":1}
I20260928 22:01:37.790711 16716 tablet_service.cc:1467] Processing CreateTablet for tablet 55b233ea0ae94ca18a5e64b6f3d71862 (DEFAULT_TABLE table=test-table [id=82687dadb57948cda714074fabcfc16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:37.790895 16850 tablet_service.cc:1467] Processing CreateTablet for tablet 55b233ea0ae94ca18a5e64b6f3d71862 (DEFAULT_TABLE table=test-table [id=82687dadb57948cda714074fabcfc16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:37.791075 16716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b233ea0ae94ca18a5e64b6f3d71862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:37.791145 16850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b233ea0ae94ca18a5e64b6f3d71862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:37.791184 16582 tablet_service.cc:1467] Processing CreateTablet for tablet 55b233ea0ae94ca18a5e64b6f3d71862 (DEFAULT_TABLE table=test-table [id=82687dadb57948cda714074fabcfc16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:37.791495 16582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b233ea0ae94ca18a5e64b6f3d71862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:37.794128 16933 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Bootstrap starting.
I20260928 22:01:37.794826 16933 tablet_bootstrap.cc:654] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:37.795071 16934 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Bootstrap starting.
I20260928 22:01:37.795468 16933 log.cc:824] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:37.795747 16934 tablet_bootstrap.cc:654] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:37.796409 16933 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: No bootstrap required, opened a new log
I20260928 22:01:37.796482 16934 log.cc:824] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:37.796579 16933 ts_tablet_manager.cc:1397] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:01:37.796931 16935 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Bootstrap starting.
I20260928 22:01:37.797309 16934 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: No bootstrap required, opened a new log
I20260928 22:01:37.797396 16934 ts_tablet_manager.cc:1397] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:01:37.797652 16935 tablet_bootstrap.cc:654] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:37.798322 16935 log.cc:824] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:37.799258 16935 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: No bootstrap required, opened a new log
I20260928 22:01:37.799125 16933 raft_consensus.cc:348] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.799327 16933 raft_consensus.cc:374] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:37.799343 16935 ts_tablet_manager.cc:1397] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:01:37.799378 16933 raft_consensus.cc:729] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11caba988dd046a59eed75129e684717, State: Initialized, Role: FOLLOWER
I20260928 22:01:37.799345 16934 raft_consensus.cc:348] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.799513 16934 raft_consensus.cc:374] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:37.799549 16934 raft_consensus.cc:729] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab70c4ab13484a8b7380c1965fdccf, State: Initialized, Role: FOLLOWER
I20260928 22:01:37.799532 16933 consensus_queue.cc:260] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.799675 16934 consensus_queue.cc:260] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.799837 16933 ts_tablet_manager.cc:1428] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:01:37.799957 16916 heartbeater.cc:503] Master 127.2.165.126:45199 was elected leader, sending a full tablet report...
I20260928 22:01:37.800056 16934 ts_tablet_manager.cc:1428] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:01:37.800168 16782 heartbeater.cc:503] Master 127.2.165.126:45199 was elected leader, sending a full tablet report...
I20260928 22:01:37.801839 16935 raft_consensus.cc:348] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.802004 16935 raft_consensus.cc:374] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:37.802038 16935 raft_consensus.cc:729] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a540c44e67046878924c4905e0bee01, State: Initialized, Role: FOLLOWER
I20260928 22:01:37.802173 16935 consensus_queue.cc:260] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.802524 16935 ts_tablet_manager.cc:1428] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:01:37.802542 16648 heartbeater.cc:503] Master 127.2.165.126:45199 was elected leader, sending a full tablet report...
W20260928 22:01:37.852473 16783 tablet.cc:2369] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:01:37.900110 16939 raft_consensus.cc:482] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:01:37.900257 16939 raft_consensus.cc:504] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.900797 16939 leader_election.cc:288] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557), abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615)
I20260928 22:01:37.904086 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.900961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42615 (local address 127.2.165.67:58257)
0928 22:01:37.901015 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.901027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.901071 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.901315 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.901316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.901318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:37.901412 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.901414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.902143 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.902144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.902936 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.902938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.903764 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.903764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.903862 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.903865 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.903948 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.903992 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:01:37.904115 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.901034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:58257 (local address 127.2.165.66:42615)
0928 22:01:37.901172 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.901174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.901185 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:01:37.901222 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.901224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.901234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.901303 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:37.901498 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.902047 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.903065 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.903698 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.903768 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.903777 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.903972 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.904018 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 22:01:37.904188 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.901094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:44195 (local address 127.2.165.65:39557)
0928 22:01:37.901171 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.901173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.901296 (+ 123us) server_negotiation.cc:407] Connection header received
0928 22:01:37.901326 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.901327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.901337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.901411 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:37.901644 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.902149 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.903235 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.903854 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.903938 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.903948 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.904082 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.904118 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 22:01:37.904475 16736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b233ea0ae94ca18a5e64b6f3d71862" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abab70c4ab13484a8b7380c1965fdccf" is_pre_election: true
I20260928 22:01:37.904522 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.900986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39557 (local address 127.2.165.67:44195)
0928 22:01:37.901202 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.901212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:37.901275 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:37.901480 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:37.901481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:37.901483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:37.901555 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.901557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.902269 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.902269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.903088 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:37.903090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.903935 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.903935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:37.903999 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.904002 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:01:37.904062 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:01:37.904093 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":125,"threads_started":1}
I20260928 22:01:37.904693 16736 raft_consensus.cc:2454] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 0.
I20260928 22:01:37.904901 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b233ea0ae94ca18a5e64b6f3d71862" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a540c44e67046878924c4905e0bee01" is_pre_election: true
I20260928 22:01:37.905019 16804 leader_election.cc:302] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11caba988dd046a59eed75129e684717, abab70c4ab13484a8b7380c1965fdccf; no voters:
I20260928 22:01:37.905061 16602 raft_consensus.cc:2454] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 0.
I20260928 22:01:37.905148 16939 raft_consensus.cc:2781] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:01:37.905236 16939 raft_consensus.cc:482] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:01:37.905262 16939 raft_consensus.cc:3037] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:37.905997 16939 raft_consensus.cc:504] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.906144 16939 leader_election.cc:288] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 election: Requested vote from peers 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557), abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615)
I20260928 22:01:37.906301 16736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b233ea0ae94ca18a5e64b6f3d71862" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abab70c4ab13484a8b7380c1965fdccf"
I20260928 22:01:37.906327 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b233ea0ae94ca18a5e64b6f3d71862" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a540c44e67046878924c4905e0bee01"
I20260928 22:01:37.906385 16736 raft_consensus.cc:3037] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:37.906396 16602 raft_consensus.cc:3037] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:37.907289 16736 raft_consensus.cc:2454] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 1.
I20260928 22:01:37.907289 16602 raft_consensus.cc:2454] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 1.
I20260928 22:01:37.907495 16804 leader_election.cc:302] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11caba988dd046a59eed75129e684717, abab70c4ab13484a8b7380c1965fdccf; no voters:
I20260928 22:01:37.907610 16939 raft_consensus.cc:2781] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:37.907804 16939 raft_consensus.cc:686] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 1 LEADER]: Becoming Leader. State: Replica: 11caba988dd046a59eed75129e684717, State: Running, Role: LEADER
I20260928 22:01:37.907965 16939 consensus_queue.cc:237] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:37.908963 16456 catalog_manager.cc:5495] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11caba988dd046a59eed75129e684717 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "11caba988dd046a59eed75129e684717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } health_report { overall_health: HEALTHY } } }
W20260928 22:01:37.917083 16649 tablet.cc:2369] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:01:37.938088 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.935399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.2.165.67:41247)
0928 22:01:37.935468 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:37.935470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:37.935501 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:01:37.935551 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:37.935552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:37.935582 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:37.935674 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:01:37.935829 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.936371 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.937182 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:37.937382 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:37.937441 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:37.937702 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:01:37.937818 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:01:37.937924 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:37.937950 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 22:01:37.937972 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:01:37.940286 16602 raft_consensus.cc:1261] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Refusing update from remote peer 11caba988dd046a59eed75129e684717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:37.940286 16736 raft_consensus.cc:1261] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Refusing update from remote peer 11caba988dd046a59eed75129e684717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:37.940762 16943 consensus_queue.cc:1035] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:37.940910 16939 consensus_queue.cc:1035] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:37.944183 16948 mvcc.cc:205] Tried to move back new op lower bound from 7334432349960343552 to 7334432349831778304. Current Snapshot: MvccSnapshot[applied={T|T < 7334432349960343552 or (T in {7334432349960343552})}]
I20260928 22:01:37.946516 16455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:37.946616 16455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:37.947515 16455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Sending DeleteTablet for 3 replicas of tablet 55b233ea0ae94ca18a5e64b6f3d71862
I20260928 22:01:37.947926 16716 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b233ea0ae94ca18a5e64b6f3d71862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:37 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:60104
I20260928 22:01:37.948060 16582 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b233ea0ae94ca18a5e64b6f3d71862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:37 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:44816
I20260928 22:01:37.948203 16850 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b233ea0ae94ca18a5e64b6f3d71862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:37 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:53230
I20260928 22:01:37.948285 16957 tablet_replica.cc:330] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: stopping tablet replica
I20260928 22:01:37.948395 16957 raft_consensus.cc:2229] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:01:37.948514 16956 tablet_replica.cc:330] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: stopping tablet replica
I20260928 22:01:37.948591 16957 raft_consensus.cc:2258] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:37.948594 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16519
I20260928 22:01:37.949640 16958 tablet_replica.cc:330] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: stopping tablet replica
I20260928 22:01:37.949757 16958 raft_consensus.cc:2229] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:01:37.949898 16958 raft_consensus.cc:2258] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:37.950062 16956 raft_consensus.cc:2229] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:01:37.950181 16956 raft_consensus.cc:2258] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:37.950673 16956 ts_tablet_manager.cc:1905] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:01:37.950801 16958 ts_tablet_manager.cc:1905] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:01:37.952260 16956 ts_tablet_manager.cc:1918] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:01:37.952366 16956 log.cc:1196] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/55b233ea0ae94ca18a5e64b6f3d71862
I20260928 22:01:37.952724 16956 ts_tablet_manager.cc:1939] T 55b233ea0ae94ca18a5e64b6f3d71862 P abab70c4ab13484a8b7380c1965fdccf: Deleting consensus metadata
I20260928 22:01:37.952777 16958 ts_tablet_manager.cc:1918] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:01:37.952888 16958 log.cc:1196] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/55b233ea0ae94ca18a5e64b6f3d71862
I20260928 22:01:37.953171 16958 ts_tablet_manager.cc:1939] T 55b233ea0ae94ca18a5e64b6f3d71862 P 11caba988dd046a59eed75129e684717: Deleting consensus metadata
I20260928 22:01:37.953203 16442 catalog_manager.cc:4841] TS abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615): tablet 55b233ea0ae94ca18a5e64b6f3d71862 (table test-table [id=82687dadb57948cda714074fabcfc16b]) successfully deleted
I20260928 22:01:37.953604 16444 catalog_manager.cc:4841] TS 11caba988dd046a59eed75129e684717 (127.2.165.67:41247): tablet 55b233ea0ae94ca18a5e64b6f3d71862 (table test-table [id=82687dadb57948cda714074fabcfc16b]) successfully deleted
W20260928 22:01:37.955521 16444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:39557 (error 108)
I20260928 22:01:37.955785 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16653
I20260928 22:01:37.956483 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:37.955944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39557 (local address 127.0.0.1:44820)
0928 22:01:37.956401 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:37.956444 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422}
W20260928 22:01:37.956630 16444 catalog_manager.cc:4584] TS 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55b233ea0ae94ca18a5e64b6f3d71862: Network error: Client connection negotiation failed: client connection to 127.2.165.65:39557: connect: Connection refused (error 111)
I20260928 22:01:37.962217 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16787
I20260928 22:01:37.967114 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16425
I20260928 22:01:37.973018 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45199
--webserver_interface=127.2.165.126
--webserver_port=45215
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:38.055471 16960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:38.055699 16960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:38.055722 16960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:38.056986 16960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:38.057030 16960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:38.057045 16960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:38.057058 16960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:38.057118 16960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:38.058599 16960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:38.058874 16960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:38.059122 16960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:38.068027 16969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.068146 16967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.068188 16966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.068495 16960 server_base.cc:1034] running on GCE node
I20260928 22:01:38.068831 16960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:38.069217 16960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:38.070379 16960 hybrid_clock.cc:648] HybridClock initialized: now 1790632898070358 us; error 39 us; skew 500 ppm
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:38.073215 16960 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:38.073730 16960 webserver.cc:460] Webserver started at http://127.2.165.126:45215/ using document root <none> and password file <none>
I20260928 22:01:38.073990 16960 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:38.074082 16960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:38.075544 16960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.076223 16976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.076500 16960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.076619 16960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "e4232c423c66416fbefd46de108e2256"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "fe905cb027944eb1499a5b27ff101489"
server_key_iv: "11d622a2fbb741daa22af7d1e89c61de"
server_key_version: "encryptionkey@0"
I20260928 22:01:38.077142 16960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:38.093536 16960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:38.095444 16960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:38.095613 16960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:38.100023 16960 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45199
I20260928 22:01:38.100057 17028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45199 every 8 connection(s)
I20260928 22:01:38.100498 16960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:38.102258 17029 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:01:38.102878 17029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Bootstrap starting.
I20260928 22:01:38.108810 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 16960
I20260928 22:01:38.109169 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:39557
--local_ip_for_outbound_sockets=127.2.165.65
--tserver_master_addrs=127.2.165.126:45199
--webserver_port=45027
--webserver_interface=127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:38.111743 17029 log.cc:824] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:38.113899 17029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:01:38.114245 17029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Bootstrap complete.
I20260928 22:01:38.116578 17029 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.116900 17029 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4232c423c66416fbefd46de108e2256, State: Initialized, Role: FOLLOWER
I20260928 22:01:38.117089 17029 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.117154 17029 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:01:38.117220 17029 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:01:38.117272 17029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:01:38.118407 17029 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.118498 17029 leader_election.cc:302] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4232c423c66416fbefd46de108e2256; no voters:
I20260928 22:01:38.118716 17029 leader_election.cc:288] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:01:38.118754 17033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:01:38.118866 17033 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 LEADER]: Becoming Leader. State: Replica: e4232c423c66416fbefd46de108e2256, State: Running, Role: LEADER
I20260928 22:01:38.119006 17029 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:38.119037 17033 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.119306 17033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4232c423c66416fbefd46de108e2256. Latest consensus state: current_term: 2 leader_uuid: "e4232c423c66416fbefd46de108e2256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } } }
I20260928 22:01:38.119376 17033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:38.119520 17035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4232c423c66416fbefd46de108e2256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } } }
I20260928 22:01:38.119616 17035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:38.119866 17038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:38.120760 17038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:38.120918 17038 catalog_manager.cc:1259] Loaded cluster ID: 07a470be8316412aad7df2d864a36f60
I20260928 22:01:38.121130 17038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:01:38.121446 17038 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 22:01:38.126124 17038 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 22:01:38.128841 17038 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:01:38.129086 17038 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Loaded TSK: 0
I20260928 22:01:38.129385 17038 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:01:38.196267 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:38.196522 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:38.196561 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:38.197897 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:38.197969 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:38.198021 17031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:01:38.199545 17031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:39557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=45027
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:38.199819 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:38.200063 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:38.206198 17056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.206352 17057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.206542 17059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.206804 17031 server_base.cc:1034] running on GCE node
I20260928 22:01:38.207020 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:38.207317 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:38.208518 17031 hybrid_clock.cc:648] HybridClock initialized: now 1790632898208496 us; error 32 us; skew 500 ppm
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:38.211726 17031 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:01:38.212287 17031 webserver.cc:460] Webserver started at http://127.2.165.65:45027/ using document root <none> and password file <none>
I20260928 22:01:38.212551 17031 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:38.212634 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:38.214449 17031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.215315 17066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.215629 17031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.215742 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "8a540c44e67046878924c4905e0bee01"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "c36490de2939ef56dac37ad0f16b8306"
server_key_iv: "3571615a87ebe89fd49b33714be833d6"
server_key_version: "encryptionkey@0"
I20260928 22:01:38.216105 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:38.234849 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:38.237327 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:38.237488 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:38.237764 17031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:38.238257 17073 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:01:38.240134 17031 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:01:38.240211 17031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:01:38.240258 17031 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:01:38.241003 17031 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:01:38.241062 17031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.241130 17073 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Bootstrap starting.
I20260928 22:01:38.248591 17031 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:39557
I20260928 22:01:38.249037 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:01:38.252384 17073 log.cc:824] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:38.252833 17180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:39557 every 8 connection(s)
I20260928 22:01:38.255039 17073 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:01:38.255234 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 17031
I20260928 22:01:38.255352 17073 tablet_bootstrap.cc:492] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Bootstrap complete.
I20260928 22:01:38.255533 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:42615
--local_ip_for_outbound_sockets=127.2.165.66
--tserver_master_addrs=127.2.165.126:45199
--webserver_port=42989
--webserver_interface=127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:38.256572 17073 ts_tablet_manager.cc:1397] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260928 22:01:38.257990 17073 raft_consensus.cc:348] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:38.258311 17073 raft_consensus.cc:729] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a540c44e67046878924c4905e0bee01, State: Initialized, Role: FOLLOWER
I20260928 22:01:38.258472 17073 consensus_queue.cc:260] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } } peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } }
I20260928 22:01:38.258774 17073 ts_tablet_manager.cc:1428] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Time spent starting tablet: real 0.002s user 0.000s sys 0.005s
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:38.265195 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.249851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:50279 (local address 127.2.165.126:45199)
0928 22:01:38.252766 (+ 2915us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.252777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.252829 (+ 52us) server_negotiation.cc:407] Connection header received
0928 22:01:38.253094 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.253100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.253148 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.253304 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:38.254032 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.254935 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.258265 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.258993 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.259118 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.261456 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:38.261472 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:38.261472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:38.261483 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:38.263183 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.263665 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.263669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.263670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.263777 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.264061 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.264062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.264062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.264355 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:38.264513 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.264856 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.264985 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2852,"thread_start_us":100,"threads_started":1}
I20260928 22:01:38.265196 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.249770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:50279)
0928 22:01:38.249959 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.250004 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.250479 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.253334 (+ 2855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.253338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.253714 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:38.253905 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.253911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.256921 (+ 3010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.256922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.258164 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.258167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.259088 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.259089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.259217 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.259809 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:38.259817 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:38.261325 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:38.263291 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.263299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.263302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.263549 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.263853 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.263854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.263854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.263983 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.264493 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:38.264499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:38.264647 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.264798 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.264992 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":114,"threads_started":1}
I20260928 22:01:38.265930 17181 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45199
I20260928 22:01:38.266094 17181 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:38.266283 17181 heartbeater.cc:511] Master 127.2.165.126:45199 requested a full tablet report, sending...
I20260928 22:01:38.267032 16993 ts_manager.cc:194] Registered new tserver with Master: 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557)
I20260928 22:01:38.268100 16993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:50279
I20260928 22:01:38.269147 17181 heartbeater.cc:503] Master 127.2.165.126:45199 was elected leader, sending a full tablet report...
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 22:01:38.277194 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.267634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39557 (local address 127.0.0.1:44832)
0928 22:01:38.267922 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.267965 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.268045 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.268362 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.268367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.268755 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:38.268815 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.268818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.270170 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.270171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.270826 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.270829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.271427 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.271428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.271536 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.272117 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:38.272123 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:38.273578 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:38.275760 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.275767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.275769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.276019 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.276269 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.276270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.276271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.276340 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.276823 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:38.276830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:38.276885 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.277004 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.277117 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260928 22:01:38.277258 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.267724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.2.165.65:39557)
0928 22:01:38.268214 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.268219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.268228 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:01:38.268261 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.268263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.268278 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.268341 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:38.268901 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.270085 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.270898 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.271356 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.271427 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.273672 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:38.273681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:38.273682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:38.273691 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:38.275651 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.276103 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.276106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.276107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.276174 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.276505 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.276507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.276507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.276745 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:38.276832 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.277055 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.277157 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":84,"threads_started":1}
I20260928 22:01:38.278137 17115 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b233ea0ae94ca18a5e64b6f3d71862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:37 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:44832
I20260928 22:01:38.278688 17190 tablet_replica.cc:330] stopping tablet replica
I20260928 22:01:38.278818 17190 raft_consensus.cc:2229] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:01:38.278904 17190 raft_consensus.cc:2258] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:38.279239 17190 ts_tablet_manager.cc:1905] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:01:38.281526 17190 ts_tablet_manager.cc:1918] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:01:38.281623 17190 log.cc:1196] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/55b233ea0ae94ca18a5e64b6f3d71862
I20260928 22:01:38.281839 17190 ts_tablet_manager.cc:1939] T 55b233ea0ae94ca18a5e64b6f3d71862 P 8a540c44e67046878924c4905e0bee01: Deleting consensus metadata
I20260928 22:01:38.282384 16980 catalog_manager.cc:4841] TS 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557): tablet 55b233ea0ae94ca18a5e64b6f3d71862 (table test-table [id=82687dadb57948cda714074fabcfc16b]) successfully deleted
W20260928 22:01:38.345711 17186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:38.345929 17186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:38.345955 17186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:38.347172 17186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:38.347220 17186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:38.347247 17186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:01:38.348732 17186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:42615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=42989
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:38.348932 17186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:38.349128 17186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:38.354393 17196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.354673 17186 server_base.cc:1034] running on GCE node
W20260928 22:01:38.354399 17198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.354416 17195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.355006 17186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:38.355291 17186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:38.356485 17186 hybrid_clock.cc:648] HybridClock initialized: now 1790632898356476 us; error 35 us; skew 500 ppm
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:38.359220 17186 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:01:38.359688 17186 webserver.cc:460] Webserver started at http://127.2.165.66:42989/ using document root <none> and password file <none>
I20260928 22:01:38.359923 17186 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:38.360008 17186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:38.361544 17186 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:01:38.362248 17205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.362422 17186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.362528 17186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "abab70c4ab13484a8b7380c1965fdccf"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "2edfb713a9b5a9e255ad1ebee87749ae"
server_key_iv: "c41c68f697b9d44f943e3e122322cd58"
server_key_version: "encryptionkey@0"
I20260928 22:01:38.362900 17186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:38.385303 17186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:38.388026 17186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:38.388234 17186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:38.388526 17186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:38.388947 17186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:38.389034 17186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.389081 17186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:38.389108 17186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.395599 17186 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:42615
I20260928 22:01:38.395637 17318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:42615 every 8 connection(s)
I20260928 22:01:38.396030 17186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:38.402770 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 17186
I20260928 22:01:38.403069 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:41247
--local_ip_for_outbound_sockets=127.2.165.67
--tserver_master_addrs=127.2.165.126:45199
--webserver_port=35707
--webserver_interface=127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:38.406404 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.396859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:59717 (local address 127.2.165.126:45199)
0928 22:01:38.396914 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.396917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.397177 (+ 260us) server_negotiation.cc:407] Connection header received
0928 22:01:38.397815 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.397817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.397827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.397883 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:38.399010 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.399573 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.400664 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.400896 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.400968 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.403356 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:38.403358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:38.403358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:38.403374 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:38.404923 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.405377 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.405379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.405379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.405450 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.405723 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.405724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.405724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.405846 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:38.405923 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.406213 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.406308 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:01:38.406610 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.396881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:59717)
0928 22:01:38.397044 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.397082 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.397660 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.397942 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.397945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.398663 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:38.398914 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.398919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.399673 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.399674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.400558 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.400561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.400980 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.400980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.401090 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.401658 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:38.401667 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:38.403238 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:38.405047 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.405055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.405058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.405297 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.405527 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.405527 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.405528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.405647 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.405914 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:38.405921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:38.406076 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.406182 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.406350 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260928 22:01:38.407147 17319 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45199
I20260928 22:01:38.407271 17319 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:38.407482 17319 heartbeater.cc:511] Master 127.2.165.126:45199 requested a full tablet report, sending...
I20260928 22:01:38.407974 16993 ts_manager.cc:194] Registered new tserver with Master: abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615)
I20260928 22:01:38.408380 16993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:59717
W20260928 22:01:38.488971 17322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:38.489204 17322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:38.489230 17322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:38.490417 17322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:38.490459 17322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:38.490492 17322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:01:38.491931 17322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:41247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=35707
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:38.492141 17322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:38.492327 17322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:38.498028 17330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.498086 17327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.498027 17328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.498240 17322 server_base.cc:1034] running on GCE node
I20260928 22:01:38.498631 17322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:38.498898 17322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:38.500061 17322 hybrid_clock.cc:648] HybridClock initialized: now 1790632898500057 us; error 58 us; skew 500 ppm
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:38.502777 17322 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:01:38.503278 17322 webserver.cc:460] Webserver started at http://127.2.165.67:35707/ using document root <none> and password file <none>
I20260928 22:01:38.503508 17322 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:38.503597 17322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:38.504987 17322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.505677 17337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.505856 17322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:01:38.505939 17322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "11caba988dd046a59eed75129e684717"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "9549a4dbe2d267127475e5b5fe84e351"
server_key_iv: "80c69cb0852db7b4db548f22aaaeb962"
server_key_version: "encryptionkey@0"
I20260928 22:01:38.506268 17322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:38.512948 17322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:38.515021 17322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:38.515136 17322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:38.515318 17322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:38.515650 17322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:38.515687 17322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.515709 17322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:38.515722 17322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.522324 17322 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:41247
I20260928 22:01:38.522471 17450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:41247 every 8 connection(s)
I20260928 22:01:38.522781 17322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:01:38.528298 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 17322
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:38.532778 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.523581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:50959 (local address 127.2.165.126:45199)
0928 22:01:38.523653 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.523655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.523874 (+ 219us) server_negotiation.cc:407] Connection header received
0928 22:01:38.524360 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.524361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.524373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.524454 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:38.525490 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.525994 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.527010 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.527205 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.527264 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.529662 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:38.529664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:38.529664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:38.529671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:38.531115 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.531635 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.531636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.531637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.531726 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.531992 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.531994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.531994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.532139 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:38.532226 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.532547 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.532634 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:01:38.532922 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.523576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:50959)
0928 22:01:38.523743 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.523779 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.524252 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.524509 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.524512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.525178 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:38.525377 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.525382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.526080 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.526081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.526921 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.526924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.527283 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.527284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.527385 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.527921 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:38.527930 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:38.529563 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:38.531224 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.531232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.531234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.531556 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.531805 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:38.531806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:38.531806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:38.531921 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:38.532222 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:38.532227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:38.532348 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.532493 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.532626 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260928 22:01:38.533434 17451 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45199
I20260928 22:01:38.533548 17451 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:38.533723 17451 heartbeater.cc:511] Master 127.2.165.126:45199 requested a full tablet report, sending...
I20260928 22:01:38.534080 16993 ts_manager.cc:194] Registered new tserver with Master: 11caba988dd046a59eed75129e684717 (127.2.165.67:41247)
I20260928 22:01:38.534459 16993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:50959
I20260928 22:01:38.535599 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.528841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.2.165.126:45199)
0928 22:01:38.528993 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.528995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.529009 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:01:38.529066 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.529067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.529078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.529154 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:38.529718 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.530206 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.530923 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.531139 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.531216 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.532758 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:38.532759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:38.532760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:38.532766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:38.534368 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.534848 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.534849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.534849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.534914 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:38.535107 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:38.535108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:38.535109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:38.535233 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:38.535397 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.535438 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.535520 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260928 22:01:38.536305 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:01:38.539000 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.536713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.2.165.126:45199)
0928 22:01:38.536750 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.536751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.536916 (+ 165us) server_negotiation.cc:407] Connection header received
0928 22:01:38.536951 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.536952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.536979 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.537048 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:01:38.537133 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.537632 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.538342 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.538568 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.538625 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.538764 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:01:38.538852 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:01:38.538908 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.538937 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.538954 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:01:38.539300 16993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:01:38.541350 16993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:01:38.543989 17115 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0eabca0c044350a35aacc3df9a385d (DEFAULT_TABLE table=test-table [id=857157c4e9cf4c3e9c37576cd72bf03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:38.544212 17115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0eabca0c044350a35aacc3df9a385d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:38.546069 17073 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Bootstrap starting.
I20260928 22:01:38.546384 17073 tablet_bootstrap.cc:654] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:38.547345 17073 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: No bootstrap required, opened a new log
I20260928 22:01:38.547411 17073 ts_tablet_manager.cc:1397] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:38.547590 17073 raft_consensus.cc:348] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.547678 17073 raft_consensus.cc:374] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:38.547726 17073 raft_consensus.cc:729] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a540c44e67046878924c4905e0bee01, State: Initialized, Role: FOLLOWER
I20260928 22:01:38.547775 17073 consensus_queue.cc:260] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.547861 17073 ts_tablet_manager.cc:1428] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 22:01:38.548189 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.543740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.2.165.67:41247)
0928 22:01:38.544776 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.544780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.544790 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:01:38.544827 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.544828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.544878 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.544991 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:38.545143 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.545937 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.546957 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.547654 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.547744 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.547810 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.548073 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.548116 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":88,"threads_started":1}
I20260928 22:01:38.548178 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.543530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:41247 (local address 127.0.0.1:53242)
0928 22:01:38.543725 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.543737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.543795 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.545006 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.545008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.545010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:38.545066 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.545067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.546048 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.546049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.546859 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.546861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.547849 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.547850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.547934 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.547960 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.548051 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.548100 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":119,"threads_started":1}
I20260928 22:01:38.548591 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.543464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42615 (local address 127.0.0.1:60110)
0928 22:01:38.543535 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.543549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.543593 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.544088 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.544089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.544090 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:38.544140 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.544142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.545533 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.545534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.546884 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.546888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.548357 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.548358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.548440 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.548445 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.548485 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.548540 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 22:01:38.549037 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.543619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.2.165.66:42615)
0928 22:01:38.543904 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.543908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.543919 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:01:38.543960 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.543962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.543978 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.544084 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:38.544200 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.545420 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.547219 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.548233 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.548778 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.548856 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.548899 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.548947 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":128,"threads_started":1}
I20260928 22:01:38.549306 17385 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0eabca0c044350a35aacc3df9a385d (DEFAULT_TABLE table=test-table [id=857157c4e9cf4c3e9c37576cd72bf03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:38.549614 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0eabca0c044350a35aacc3df9a385d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:38.549813 17253 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0eabca0c044350a35aacc3df9a385d (DEFAULT_TABLE table=test-table [id=857157c4e9cf4c3e9c37576cd72bf03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:38.550112 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0eabca0c044350a35aacc3df9a385d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:38.552500 17461 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Bootstrap starting.
I20260928 22:01:38.553198 17462 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Bootstrap starting.
I20260928 22:01:38.553409 17461 tablet_bootstrap.cc:654] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:38.553850 17462 tablet_bootstrap.cc:654] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:38.553992 17461 log.cc:824] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:38.554419 17462 log.cc:824] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:38.554842 17461 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: No bootstrap required, opened a new log
I20260928 22:01:38.554943 17461 ts_tablet_manager.cc:1397] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:01:38.555250 17462 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: No bootstrap required, opened a new log
I20260928 22:01:38.555339 17462 ts_tablet_manager.cc:1397] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:01:38.556732 17461 raft_consensus.cc:348] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.556918 17461 raft_consensus.cc:374] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:38.556946 17461 raft_consensus.cc:729] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11caba988dd046a59eed75129e684717, State: Initialized, Role: FOLLOWER
I20260928 22:01:38.557096 17461 consensus_queue.cc:260] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.557075 17462 raft_consensus.cc:348] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.557261 17462 raft_consensus.cc:374] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:38.557310 17462 raft_consensus.cc:729] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab70c4ab13484a8b7380c1965fdccf, State: Initialized, Role: FOLLOWER
I20260928 22:01:38.557416 17451 heartbeater.cc:503] Master 127.2.165.126:45199 was elected leader, sending a full tablet report...
I20260928 22:01:38.557432 17462 consensus_queue.cc:260] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.557492 17461 ts_tablet_manager.cc:1428] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:01:38.557700 17462 ts_tablet_manager.cc:1428] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:01:38.557730 17319 heartbeater.cc:503] Master 127.2.165.126:45199 was elected leader, sending a full tablet report...
I20260928 22:01:38.558306 17465 raft_consensus.cc:482] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:01:38.558406 17465 raft_consensus.cc:504] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.558786 17465 leader_election.cc:288] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615), 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557)
I20260928 22:01:38.561986 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.559172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:58959 (local address 127.2.165.65:39557)
0928 22:01:38.559219 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.559221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.559230 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:01:38.559256 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.559257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.559282 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.559343 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:38.559496 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.560063 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.561198 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.561638 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.561740 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.561816 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.561879 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.561918 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:01:38.562005 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.558940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39557 (local address 127.2.165.67:58959)
0928 22:01:38.559080 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.559091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.559161 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.559347 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.559348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.559349 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:38.559416 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.559417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.560161 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.560162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.561104 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.561106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.561741 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.561742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.561807 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.561812 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.561872 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.561914 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":30,"thread_start_us":94,"threads_started":1}
I20260928 22:01:38.562100 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.558918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42615 (local address 127.2.165.67:60077)
0928 22:01:38.558957 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.558972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.559019 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.559289 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:38.559290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:38.559292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:38.559359 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.559360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.560203 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.560204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.561128 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:38.561130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.561796 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.561797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:38.561897 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.561900 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:01:38.561981 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:01:38.562025 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":29}
I20260928 22:01:38.562103 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.559118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:60077 (local address 127.2.165.66:42615)
0928 22:01:38.559181 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.559184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.559192 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:01:38.559218 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.559220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.559230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.559284 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:38.559434 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.560113 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.561253 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.561691 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.561796 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.561806 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.561991 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.562030 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:01:38.562268 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0eabca0c044350a35aacc3df9a385d" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a540c44e67046878924c4905e0bee01" is_pre_election: true
I20260928 22:01:38.562422 17121 raft_consensus.cc:2454] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 0.
I20260928 22:01:38.562422 17273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0eabca0c044350a35aacc3df9a385d" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abab70c4ab13484a8b7380c1965fdccf" is_pre_election: true
I20260928 22:01:38.562534 17273 raft_consensus.cc:2454] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 0.
I20260928 22:01:38.562772 17341 leader_election.cc:302] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11caba988dd046a59eed75129e684717, 8a540c44e67046878924c4905e0bee01; no voters:
I20260928 22:01:38.562939 17465 raft_consensus.cc:2781] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:01:38.562999 17465 raft_consensus.cc:482] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:01:38.563023 17465 raft_consensus.cc:3037] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:38.563819 17465 raft_consensus.cc:504] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.563969 17465 leader_election.cc:288] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 election: Requested vote from peers abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615), 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557)
I20260928 22:01:38.564123 17273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0eabca0c044350a35aacc3df9a385d" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abab70c4ab13484a8b7380c1965fdccf"
I20260928 22:01:38.564164 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0eabca0c044350a35aacc3df9a385d" candidate_uuid: "11caba988dd046a59eed75129e684717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a540c44e67046878924c4905e0bee01"
I20260928 22:01:38.564210 17273 raft_consensus.cc:3037] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:38.564231 17121 raft_consensus.cc:3037] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:38.565013 17121 raft_consensus.cc:2454] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 1.
I20260928 22:01:38.565114 17273 raft_consensus.cc:2454] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11caba988dd046a59eed75129e684717 in term 1.
I20260928 22:01:38.565197 17341 leader_election.cc:302] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11caba988dd046a59eed75129e684717, 8a540c44e67046878924c4905e0bee01; no voters:
I20260928 22:01:38.565296 17465 raft_consensus.cc:2781] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:38.565452 17465 raft_consensus.cc:686] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 1 LEADER]: Becoming Leader. State: Replica: 11caba988dd046a59eed75129e684717, State: Running, Role: LEADER
I20260928 22:01:38.565565 17465 consensus_queue.cc:237] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.566440 16993 catalog_manager.cc:5495] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11caba988dd046a59eed75129e684717 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "11caba988dd046a59eed75129e684717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } health_report { overall_health: UNKNOWN } } }
I20260928 22:01:38.573526 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.570886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.2.165.67:41247)
0928 22:01:38.571095 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:38.571097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:38.571108 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:01:38.571149 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:38.571150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:38.571166 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:38.571243 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:01:38.571386 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.571865 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.572613 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:38.572857 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:38.572934 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:38.573064 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:01:38.573175 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:01:38.573243 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:38.573359 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 22:01:38.573380 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:01:38.575685 17273 raft_consensus.cc:1261] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Refusing update from remote peer 11caba988dd046a59eed75129e684717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:38.575701 17121 raft_consensus.cc:1261] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Refusing update from remote peer 11caba988dd046a59eed75129e684717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:38.576051 17465 consensus_queue.cc:1035] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:38.576265 17468 consensus_queue.cc:1035] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:38.578523 17470 mvcc.cc:205] Tried to move back new op lower bound from 7334432352562946048 to 7334432352525316096. Current Snapshot: MvccSnapshot[applied={T|T < 7334432352562946048}]
I20260928 22:01:38.578903 17471 mvcc.cc:205] Tried to move back new op lower bound from 7334432352562946048 to 7334432352525316096. Current Snapshot: MvccSnapshot[applied={T|T < 7334432352562946048}]
I20260928 22:01:38.579238 17472 mvcc.cc:205] Tried to move back new op lower bound from 7334432352562946048 to 7334432352525316096. Current Snapshot: MvccSnapshot[applied={T|T < 7334432352562946048}]
I20260928 22:01:38.581007 16993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:38.581130 16993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:38.582156 16993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Sending DeleteTablet for 3 replicas of tablet 1a0eabca0c044350a35aacc3df9a385d
I20260928 22:01:38.582525 17253 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a0eabca0c044350a35aacc3df9a385d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:38 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:60110
I20260928 22:01:38.582597 17115 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a0eabca0c044350a35aacc3df9a385d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:38 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:44832
I20260928 22:01:38.582608 17385 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a0eabca0c044350a35aacc3df9a385d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:38 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:53242
I20260928 22:01:38.582772 17190 tablet_replica.cc:330] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: stopping tablet replica
I20260928 22:01:38.582832 17480 tablet_replica.cc:330] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: stopping tablet replica
I20260928 22:01:38.582845 17190 raft_consensus.cc:2229] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:01:38.582851 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 17031
I20260928 22:01:38.582911 17190 raft_consensus.cc:2258] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:38.583626 17480 raft_consensus.cc:2229] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:01:38.583775 17480 raft_consensus.cc:2258] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:38.584252 17480 ts_tablet_manager.cc:1905] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:01:38.586241 17480 ts_tablet_manager.cc:1918] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:01:38.586342 17480 log.cc:1196] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/1a0eabca0c044350a35aacc3df9a385d
I20260928 22:01:38.586639 17480 ts_tablet_manager.cc:1939] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Deleting consensus metadata
I20260928 22:01:38.587132 16980 catalog_manager.cc:4841] TS 11caba988dd046a59eed75129e684717 (127.2.165.67:41247): tablet 1a0eabca0c044350a35aacc3df9a385d (table test-table [id=857157c4e9cf4c3e9c37576cd72bf03a]) successfully deleted
W20260928 22:01:38.588691 16980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:39557 (error 108)
I20260928 22:01:38.588999 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 17186
I20260928 22:01:38.589713 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.589119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39557 (local address 127.0.0.1:44836)
0928 22:01:38.589635 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.589678 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489}
W20260928 22:01:38.592324 16980 catalog_manager.cc:4584] TS 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a0eabca0c044350a35aacc3df9a385d: Network error: Client connection negotiation failed: client connection to 127.2.165.65:39557: connect: Connection refused (error 111)
I20260928 22:01:38.596336 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 17322
I20260928 22:01:38.596385 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.596243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42615 (local address 127.0.0.1:60122)
0928 22:01:38.596304 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.596342 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.66:42615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:01:38.601801 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 16960
I20260928 22:01:38.607687 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45199
--webserver_interface=127.2.165.126
--webserver_port=45215
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:38.695205 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:38.695521 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:38.695565 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:38.696870 17482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:38.696954 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:38.696979 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:38.697009 17482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:38.697057 17482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:38.698823 17482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=45215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:38.699128 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:38.699383 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:38.704744 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.704787 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.704923 17482 server_base.cc:1034] running on GCE node
W20260928 22:01:38.704744 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.705412 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:38.705742 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:38.706920 17482 hybrid_clock.cc:648] HybridClock initialized: now 1790632898706891 us; error 43 us; skew 500 ppm
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:38.709697 17482 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:38.710240 17482 webserver.cc:460] Webserver started at http://127.2.165.126:45215/ using document root <none> and password file <none>
I20260928 22:01:38.710505 17482 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:38.710558 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:38.711988 17482 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:38.712812 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.713013 17482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.713116 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "e4232c423c66416fbefd46de108e2256"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "fe905cb027944eb1499a5b27ff101489"
server_key_iv: "11d622a2fbb741daa22af7d1e89c61de"
server_key_version: "encryptionkey@0"
I20260928 22:01:38.713495 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:38.723156 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:38.725564 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:38.725729 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:38.729851 17482 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45199
I20260928 22:01:38.729902 17549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45199 every 8 connection(s)
I20260928 22:01:38.730290 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:38.731863 17550 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:01:38.732554 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Bootstrap starting.
I20260928 22:01:38.732949 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 17482
I20260928 22:01:38.733230 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:39557
--local_ip_for_outbound_sockets=127.2.165.65
--tserver_master_addrs=127.2.165.126:45199
--webserver_port=45027
--webserver_interface=127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:38.741743 17550 log.cc:824] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:38.744082 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:01:38.744428 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256: Bootstrap complete.
I20260928 22:01:38.746778 17550 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.747007 17550 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4232c423c66416fbefd46de108e2256, State: Initialized, Role: FOLLOWER
I20260928 22:01:38.747184 17550 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.747256 17550 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:01:38.747304 17550 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:01:38.747352 17550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:01:38.748411 17550 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.748502 17550 leader_election.cc:302] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4232c423c66416fbefd46de108e2256; no voters:
I20260928 22:01:38.748747 17550 leader_election.cc:288] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:01:38.748775 17554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:01:38.748900 17554 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [term 3 LEADER]: Becoming Leader. State: Replica: e4232c423c66416fbefd46de108e2256, State: Running, Role: LEADER
I20260928 22:01:38.749003 17554 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } }
I20260928 22:01:38.749070 17550 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:38.749264 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e4232c423c66416fbefd46de108e2256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } } }
I20260928 22:01:38.749351 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:38.749259 17554 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4232c423c66416fbefd46de108e2256. Latest consensus state: current_term: 3 leader_uuid: "e4232c423c66416fbefd46de108e2256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4232c423c66416fbefd46de108e2256" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45199 } } }
I20260928 22:01:38.749534 17554 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4232c423c66416fbefd46de108e2256 [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:38.749657 17558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:38.750550 17558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:38.750679 17558 catalog_manager.cc:1259] Loaded cluster ID: 07a470be8316412aad7df2d864a36f60
I20260928 22:01:38.750736 17558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 22:01:38.751228 17558 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdd93db19ad google::LogMessage::Fail() at ??:0
@ 0x7fdd93db488a google::LogMessage::SendToLog() at ??:0
@ 0x7fdd93db14de google::LogMessage::Flush() at ??:0
@ 0x7fdd93db2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdd9bcd20f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdd9bcdb8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdd95c7f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdd95c74954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdd96ad86db start_thread at ??:0
@ 0x7fdd928af61f clone at ??:0
W20260928 22:01:38.832034 17552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:38.832245 17552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:38.832270 17552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:38.833520 17552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:38.833566 17552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:38.833595 17552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:01:38.835074 17552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:39557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=45027
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:38.835338 17552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:38.835636 17552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:38.841306 17589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.841517 17552 server_base.cc:1034] running on GCE node
W20260928 22:01:38.841329 17587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.841622 17586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.841849 17552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:38.842118 17552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:38.843282 17552 hybrid_clock.cc:648] HybridClock initialized: now 1790632898843263 us; error 35 us; skew 500 ppm
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:38.845968 17552 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:01:38.846516 17552 webserver.cc:460] Webserver started at http://127.2.165.65:45027/ using document root <none> and password file <none>
I20260928 22:01:38.846765 17552 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:38.846813 17552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:38.848223 17552 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:38.848948 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.849097 17552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:38.849182 17552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "8a540c44e67046878924c4905e0bee01"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "c36490de2939ef56dac37ad0f16b8306"
server_key_iv: "3571615a87ebe89fd49b33714be833d6"
server_key_version: "encryptionkey@0"
I20260928 22:01:38.849548 17552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:38.863827 17552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:38.866184 17552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:38.866336 17552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:38.866607 17552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:38.867091 17603 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:01:38.868592 17552 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:01:38.868647 17552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:01:38.868716 17552 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:01:38.869470 17552 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:01:38.869515 17552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:01:38.869568 17603 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Bootstrap starting.
I20260928 22:01:38.877411 17552 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:39557
I20260928 22:01:38.877969 17552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:01:38.878459 17710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:39557 every 8 connection(s)
I20260928 22:01:38.878683 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 17552
I20260928 22:01:38.878980 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:42615
--local_ip_for_outbound_sockets=127.2.165.66
--tserver_master_addrs=127.2.165.126:45199
--webserver_port=42989
--webserver_interface=127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:01:38.881443 17603 log.cc:824] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:38.885383 17603 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:01:38.885743 17603 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Bootstrap complete.
I20260928 22:01:38.887112 17603 ts_tablet_manager.cc:1397] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20260928 22:01:38.889106 17603 raft_consensus.cc:348] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.889539 17603 raft_consensus.cc:729] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a540c44e67046878924c4905e0bee01, State: Initialized, Role: FOLLOWER
I20260928 22:01:38.889746 17603 consensus_queue.cc:260] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:38.890131 17603 ts_tablet_manager.cc:1428] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260928 22:01:38.946153 17713 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:01:38.879137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:43827)
0928 22:01:38.881756 (+ 2619us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.881805 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:38.882581 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:38.945904 (+ 63323us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2570,"thread_start_us":107,"threads_started":1}
W20260928 22:01:38.946473 17711 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:45199 (0 consecutive failures): Network error: Failed to ping master at 127.2.165.126:45199: Client connection negotiation failed: client connection to 127.2.165.126:45199: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:01:38.946928 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.946778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:36587)
0928 22:01:38.946834 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.946894 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 22:01:38.947360 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:38.947255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:47337)
0928 22:01:38.947306 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:38.947335 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:01:38.947533 17711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:01:38.966630 17714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:38.966849 17714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:38.966874 17714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:38.968084 17714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:38.968132 17714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:38.968159 17714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:01:38.969718 17714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:42615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=42989
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:38.969993 17714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:38.970247 17714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:38.975988 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.975951 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:38.975951 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:38.976351 17714 server_base.cc:1034] running on GCE node
I20260928 22:01:38.976661 17714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:38.977005 17714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:38.978154 17714 hybrid_clock.cc:648] HybridClock initialized: now 1790632898978132 us; error 37 us; skew 500 ppm
Sep 28 22:01:38 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:38.980849 17714 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:01:38.981326 17714 webserver.cc:460] Webserver started at http://127.2.165.66:42989/ using document root <none> and password file <none>
I20260928 22:01:38.981599 17714 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:38.981648 17714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:38.983044 17714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:38.983803 17731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:38.983986 17714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:01:38.984089 17714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "abab70c4ab13484a8b7380c1965fdccf"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "2edfb713a9b5a9e255ad1ebee87749ae"
server_key_iv: "c41c68f697b9d44f943e3e122322cd58"
server_key_version: "encryptionkey@0"
I20260928 22:01:38.984483 17714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:38.992749 17714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:38.994676 17714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:38.994840 17714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:38.995074 17714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:38.995515 17738 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:01:38.997157 17714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:01:38.997246 17714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:01:38.997296 17714 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:01:38.997960 17714 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:01:38.998025 17714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:01:38.998111 17738 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Bootstrap starting.
I20260928 22:01:39.005601 17714 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:42615
I20260928 22:01:39.005722 17845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:42615 every 8 connection(s)
I20260928 22:01:39.006088 17714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:01:39.007428 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.006885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:38769)
0928 22:01:39.007089 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.007208 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
W20260928 22:01:39.007771 17846 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:45199 (0 consecutive failures): Network error: Failed to ping master at 127.2.165.126:45199: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
I20260928 22:01:39.008136 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.008015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:59247)
0928 22:01:39.008072 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.008110 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:01:39.008723 17738 log.cc:824] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:39.008785 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.008601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:54371)
0928 22:01:39.008659 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.008723 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 22:01:39.008926 17846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:01:39.010942 17738 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:01:39.011206 17738 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Bootstrap complete.
I20260928 22:01:39.012125 17738 ts_tablet_manager.cc:1397] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260928 22:01:39.013538 17738 raft_consensus.cc:348] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:39.013803 17738 raft_consensus.cc:729] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab70c4ab13484a8b7380c1965fdccf, State: Initialized, Role: FOLLOWER
I20260928 22:01:39.013947 17738 consensus_queue.cc:260] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:39.014247 17738 ts_tablet_manager.cc:1428] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260928 22:01:39.014650 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.014510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:40217)
0928 22:01:39.014582 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.014623 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 22:01:39.015022 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.014937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:60605)
0928 22:01:39.014978 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.015003 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:01:39.015308 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 17714
I20260928 22:01:39.015636 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:41247
--local_ip_for_outbound_sockets=127.2.165.67
--tserver_master_addrs=127.2.165.126:45199
--webserver_port=35707
--webserver_interface=127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:39.098317 17851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:39.098558 17851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:39.098584 17851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:39.099871 17851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:39.099920 17851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:39.099949 17851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:01:39.101658 17851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:41247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=35707
--tserver_master_addrs=127.2.165.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:39.101939 17851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:39.102181 17851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:39.107276 17856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:39.107291 17859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:39.107308 17851 server_base.cc:1034] running on GCE node
W20260928 22:01:39.107291 17857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:39.107784 17851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:39.108021 17851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:39.109170 17851 hybrid_clock.cc:648] HybridClock initialized: now 1790632899109156 us; error 28 us; skew 500 ppm
Sep 28 22:01:39 dist-test-slave-3t36 krb5kdc[16412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632899, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:39.111847 17851 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:01:39.112309 17851 webserver.cc:460] Webserver started at http://127.2.165.67:35707/ using document root <none> and password file <none>
I20260928 22:01:39.112553 17851 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:39.112600 17851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:39.114027 17851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:39.114652 17866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:39.114842 17851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:01:39.114949 17851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "11caba988dd046a59eed75129e684717"
format_stamp: "Formatted at 2026-09-28 22:01:37 on dist-test-slave-3t36"
server_key: "9549a4dbe2d267127475e5b5fe84e351"
server_key_iv: "80c69cb0852db7b4db548f22aaaeb962"
server_key_version: "encryptionkey@0"
I20260928 22:01:39.115295 17851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:39.124480 17851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:39.126533 17851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:39.126724 17851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:39.127044 17851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:39.127426 17851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:39.127480 17851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:39.127519 17851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:39.127538 17851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:39.134224 17851 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:41247
I20260928 22:01:39.134258 17979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:41247 every 8 connection(s)
I20260928 22:01:39.134596 17851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:01:39.135733 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.135271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:57157)
0928 22:01:39.135437 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.135585 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260928 22:01:39.136161 17980 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:45199 (0 consecutive failures): Network error: Failed to ping master at 127.2.165.126:45199: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
I20260928 22:01:39.136533 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.136370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:38879)
0928 22:01:39.136421 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.136454 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:01:39.136976 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.136837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:44359)
0928 22:01:39.136918 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.136950 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 22:01:39.137153 17980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:01:39.140933 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 17851
I20260928 22:01:39.948239 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:39.947989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:48359)
0928 22:01:39.948118 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:39.948164 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260928 22:01:40.015833 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.015544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:42363)
0928 22:01:40.015737 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.015781 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260928 22:01:40.137790 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.137578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:43929)
0928 22:01:40.137702 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.137739 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 22:01:40.170828 17987 raft_consensus.cc:482] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:01:40.171005 17987 raft_consensus.cc:504] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:40.171428 17987 leader_election.cc:288] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11caba988dd046a59eed75129e684717 (127.2.165.67:41247), 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557)
Sep 28 22:01:40 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
Sep 28 22:01:40 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632898, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.67@KRBTEST.COM
I20260928 22:01:40.182523 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.171759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39557 (local address 127.2.165.66:42189)
0928 22:01:40.171860 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.171871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:40.171996 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:40.172475 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:40.172480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:40.172945 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:40.173210 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:40.173216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.174295 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.174296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:40.175275 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:40.175277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.175883 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.175884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:40.175998 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:40.176656 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:40.176662 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:40.178373 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:40.180589 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:40.180598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:40.180600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:40.180948 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:40.181249 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:40.181250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:40.181251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:40.181405 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:40.181949 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:40.181956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:40.182101 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 22:01:40.182204 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 22:01:40.182416 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":58,"thread_start_us":52,"threads_started":1}
I20260928 22:01:40.182552 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.171869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:42189 (local address 127.2.165.65:39557)
0928 22:01:40.172153 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:40.172157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:40.172164 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:01:40.172296 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:40.172298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:40.172328 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:40.172421 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:40.173320 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.174190 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.175372 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.175788 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.175913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:40.178545 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:40.178550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:40.178550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:40.178564 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:40.180488 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:40.181030 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:40.181034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:40.181034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:40.181168 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:40.181530 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:40.181532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:40.181532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:40.181838 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:40.181903 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:40.182281 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 22:01:40.182433 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":66,"threads_started":1}
I20260928 22:01:40.182564 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.171750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:56019 (local address 127.2.165.67:41247)
0928 22:01:40.171932 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:40.171943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:40.171975 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:01:40.172560 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:40.172562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:40.172638 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:40.172822 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:40.173330 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.174196 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.175375 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.175787 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.175907 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:40.178866 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:40.178870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:40.178871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:40.178889 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:40.180754 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:40.181158 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:40.181160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:40.181160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:40.181304 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:40.181670 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:40.181671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:40.181671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:40.181960 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:40.182043 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:40.182274 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 22:01:40.182404 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260928 22:01:40.182960 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.171623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:41247 (local address 127.2.165.66:56019)
0928 22:01:40.171713 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.171749 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:40.171987 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:40.172804 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:40.172805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:40.173050 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:40.173210 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:40.173216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.174312 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.174313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:40.175274 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:40.175277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:40.175881 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:40.175882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:40.175998 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:40.176597 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:40.176636 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:40.178756 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:40.180840 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:40.180841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:40.180842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:40.181054 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:40.181495 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:40.181496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:40.181496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:40.181577 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:40.182035 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:40.182036 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:40.182101 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 22:01:40.182205 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 22:01:40.182499 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":153}
I20260928 22:01:40.183156 17665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0eabca0c044350a35aacc3df9a385d" candidate_uuid: "abab70c4ab13484a8b7380c1965fdccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8a540c44e67046878924c4905e0bee01" is_pre_election: true
I20260928 22:01:40.183347 17665 raft_consensus.cc:2454] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abab70c4ab13484a8b7380c1965fdccf in term 1.
I20260928 22:01:40.183661 17735 leader_election.cc:302] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a540c44e67046878924c4905e0bee01, abab70c4ab13484a8b7380c1965fdccf; no voters:
I20260928 22:01:40.183790 17987 raft_consensus.cc:2781] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 22:01:40.183851 17987 raft_consensus.cc:482] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260928 22:01:40.183831 17735 leader_election.cc:341] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11caba988dd046a59eed75129e684717 (127.2.165.67:41247): Not found: Tablet not found: 1a0eabca0c044350a35aacc3df9a385d
I20260928 22:01:40.183889 17987 raft_consensus.cc:3037] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:01:40.185191 17987 raft_consensus.cc:504] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:40.185351 17987 leader_election.cc:288] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [CANDIDATE]: Term 2 election: Requested vote from peers 11caba988dd046a59eed75129e684717 (127.2.165.67:41247), 8a540c44e67046878924c4905e0bee01 (127.2.165.65:39557)
I20260928 22:01:40.185616 17665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a0eabca0c044350a35aacc3df9a385d" candidate_uuid: "abab70c4ab13484a8b7380c1965fdccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8a540c44e67046878924c4905e0bee01"
I20260928 22:01:40.185708 17665 raft_consensus.cc:3037] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 1 FOLLOWER]: Advancing to term 2
W20260928 22:01:40.185708 17735 leader_election.cc:341] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 11caba988dd046a59eed75129e684717 (127.2.165.67:41247): Not found: Tablet not found: 1a0eabca0c044350a35aacc3df9a385d
I20260928 22:01:40.186803 17665 raft_consensus.cc:2454] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abab70c4ab13484a8b7380c1965fdccf in term 2.
I20260928 22:01:40.187026 17735 leader_election.cc:302] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a540c44e67046878924c4905e0bee01, abab70c4ab13484a8b7380c1965fdccf; no voters: 11caba988dd046a59eed75129e684717
I20260928 22:01:40.187124 17987 raft_consensus.cc:2781] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:01:40.187273 17987 raft_consensus.cc:686] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [term 2 LEADER]: Becoming Leader. State: Replica: abab70c4ab13484a8b7380c1965fdccf, State: Running, Role: LEADER
I20260928 22:01:40.187390 17987 consensus_queue.cc:237] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:40.187701 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.187597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:33691)
0928 22:01:40.187633 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.187675 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 22:01:40.627921 17735 consensus_peers.cc:487] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf -> Peer 11caba988dd046a59eed75129e684717 (127.2.165.67:41247): Couldn't send request to peer 11caba988dd046a59eed75129e684717. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a0eabca0c044350a35aacc3df9a385d. This is attempt 1: this message will repeat every 5th retry.
I20260928 22:01:40.647809 17665 raft_consensus.cc:1261] T 1a0eabca0c044350a35aacc3df9a385d P 8a540c44e67046878924c4905e0bee01 [term 2 FOLLOWER]: Refusing update from remote peer abab70c4ab13484a8b7380c1965fdccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 22:01:40.648298 17987 consensus_queue.cc:1035] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 22:01:40.649507 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.649196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:50295)
0928 22:01:40.649319 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.649399 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 22:01:40.650280 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.650170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:42659)
0928 22:01:40.650211 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.650245 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 22:01:40.650480 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:40.650328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:38331)
0928 22:01:40.650376 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:40.650401 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:01:41.046053 17998 ts_tablet_manager.cc:927] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Initiating tablet copy from peer abab70c4ab13484a8b7380c1965fdccf (127.2.165.66:42615)
I20260928 22:01:41.046334 17998 tablet_copy_client.cc:316] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: tablet copy: Beginning tablet copy session from remote peer at address 127.2.165.66:42615
Sep 28 22:01:41 dist-test-slave-3t36 krb5kdc[16412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632899, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.66@KRBTEST.COM
I20260928 22:01:41.054888 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:41.046746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:35791 (local address 127.2.165.66:42615)
0928 22:01:41.046881 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:41.046886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:41.046901 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:01:41.046971 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:41.046973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:41.046986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:41.047062 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:41.047536 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:41.048226 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:41.049126 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:41.049433 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:41.049521 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:41.051866 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:41.051874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:41.051874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:41.051881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:41.053697 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:41.054126 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:41.054131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:41.054131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:41.054193 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:41.054380 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:41.054380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:41.054381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:41.054529 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:41.054593 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:41.054747 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 22:01:41.054815 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260928 22:01:41.054890 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:41.046657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42615 (local address 127.2.165.67:35791)
0928 22:01:41.046826 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:41.046844 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:41.046915 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:41.047078 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:41.047083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:41.047381 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:41.047450 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:41.047452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:41.048312 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:41.048313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:41.049047 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:41.049050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:41.049547 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:41.049548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:41.049610 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:41.050148 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:41.050153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:41.051783 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:41.053802 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:41.053810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:41.053812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:41.054037 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:41.054260 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:41.054261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:41.054261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:41.054322 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:41.054585 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:41.054591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:41.054642 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 22:01:41.054707 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:01:41.054800 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260928 22:01:41.055397 17820 tablet_copy_service.cc:140] P abab70c4ab13484a8b7380c1965fdccf: Received BeginTabletCopySession request for tablet 1a0eabca0c044350a35aacc3df9a385d from peer 11caba988dd046a59eed75129e684717 ({username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:35791)
I20260928 22:01:41.055498 17820 tablet_copy_service.cc:161] P abab70c4ab13484a8b7380c1965fdccf: Beginning new tablet copy session on tablet 1a0eabca0c044350a35aacc3df9a385d from peer 11caba988dd046a59eed75129e684717 at {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:35791: session id = 11caba988dd046a59eed75129e684717-1a0eabca0c044350a35aacc3df9a385d
I20260928 22:01:41.056003 17820 tablet_copy_source_session.cc:205] T 1a0eabca0c044350a35aacc3df9a385d P abab70c4ab13484a8b7380c1965fdccf: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 22:01:41.057118 17998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0eabca0c044350a35aacc3df9a385d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:41.059811 17998 tablet_copy_client.cc:791] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: tablet copy: Starting download of 0 data blocks...
I20260928 22:01:41.059980 17998 tablet_copy_client.cc:655] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: tablet copy: Starting download of 1 WAL segments...
I20260928 22:01:41.060984 17998 tablet_copy_client.cc:531] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 22:01:41.061775 17998 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Bootstrap starting.
I20260928 22:01:41.063213 17998 log.cc:824] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:41.066048 17998 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:01:41.066336 17998 tablet_bootstrap.cc:492] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Bootstrap complete.
I20260928 22:01:41.066488 17998 ts_tablet_manager.cc:1397] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 22:01:41.067484 17998 raft_consensus.cc:348] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:41.067662 17998 raft_consensus.cc:729] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11caba988dd046a59eed75129e684717, State: Initialized, Role: FOLLOWER
I20260928 22:01:41.067800 17998 consensus_queue.cc:260] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abab70c4ab13484a8b7380c1965fdccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42615 } } peers { permanent_uuid: "11caba988dd046a59eed75129e684717" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 41247 } } peers { permanent_uuid: "8a540c44e67046878924c4905e0bee01" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39557 } }
I20260928 22:01:41.068076 17998 ts_tablet_manager.cc:1428] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:01:41.068428 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:41.068304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:34317)
0928 22:01:41.068359 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:41.068390 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:01:41.068442 17820 tablet_copy_service.cc:342] P abab70c4ab13484a8b7380c1965fdccf: Request end of tablet copy session 11caba988dd046a59eed75129e684717-1a0eabca0c044350a35aacc3df9a385d received from {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:35791
I20260928 22:01:41.068512 17820 tablet_copy_service.cc:433] P abab70c4ab13484a8b7380c1965fdccf: ending tablet copy session 11caba988dd046a59eed75129e684717-1a0eabca0c044350a35aacc3df9a385d on tablet 1a0eabca0c044350a35aacc3df9a385d with peer 11caba988dd046a59eed75129e684717
I20260928 22:01:41.068817 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:41.068727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:43749)
0928 22:01:41.068761 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:41.068786 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 22:01:41.512614 17934 raft_consensus.cc:1203] T 1a0eabca0c044350a35aacc3df9a385d P 11caba988dd046a59eed75129e684717 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 22:01:41.512750 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:41.512418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:59137)
0928 22:01:41.512597 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:41.512641 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":121,"threads_started":1}
I20260928 22:01:41.513233 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:41.513112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:49181)
0928 22:01:41.513167 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:41.513195 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:01:41.651117 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:41.650891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:36699)
0928 22:01:41.651033 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:41.651070 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260928 22:01:42.513409 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:42.513184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:46979)
0928 22:01:42.513317 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:42.513360 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260928 22:01:42.513841 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:42.513626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:57815)
0928 22:01:42.513739 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:42.513781 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260928 22:01:42.651978 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:42.651732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:42129)
0928 22:01:42.651876 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:42.651921 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260928 22:01:43.514504 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:43.514234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:34261)
0928 22:01:43.514395 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:43.514432 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260928 22:01:43.514537 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:43.514299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:54511)
0928 22:01:43.514446 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:43.514495 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260928 22:01:43.652810 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:43.652535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:48673)
0928 22:01:43.652714 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:43.652761 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260928 22:01:44.515280 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:44.515069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:58517)
0928 22:01:44.515201 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:44.515234 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260928 22:01:44.515309 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:44.515100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:34179)
0928 22:01:44.515226 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:44.515266 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260928 22:01:44.515893 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:44.515754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:43415)
0928 22:01:44.515827 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:44.515855 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 22:01:44.653523 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:44.653290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:51479)
0928 22:01:44.653427 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:44.653464 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260928 22:01:45.516043 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:45.515800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:43611)
0928 22:01:45.515946 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:45.515992 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260928 22:01:45.516557 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:45.516361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:36673)
0928 22:01:45.516476 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:45.516519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260928 22:01:45.654331 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:45.654043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:53553)
0928 22:01:45.654232 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:45.654282 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260928 22:01:46.517022 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:46.516792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:56181)
0928 22:01:46.516926 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:46.516974 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260928 22:01:46.517215 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:46.517005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:40819)
0928 22:01:46.517117 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:46.517156 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260928 22:01:46.655220 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:46.654970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:60527)
0928 22:01:46.655123 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:46.655168 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260928 22:01:47.517876 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:47.517602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:52695)
0928 22:01:47.517750 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:47.517798 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260928 22:01:47.517886 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:47.517669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:38515)
0928 22:01:47.517797 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:47.517833 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 22:01:47.655968 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:47.655742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:49783)
0928 22:01:47.655883 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:47.655920 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 22:01:48.518594 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:48.518410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:33803)
0928 22:01:48.518519 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:48.518549 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260928 22:01:48.518657 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:48.518392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:42259)
0928 22:01:48.518527 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:48.518587 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 22:01:48.656867 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:48.656513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:53875)
0928 22:01:48.656765 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:48.656818 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":88,"threads_started":1}
I20260928 22:01:49.519352 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:49.519112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:58887)
0928 22:01:49.519260 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:49.519303 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260928 22:01:49.519366 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:49.519109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:45195)
0928 22:01:49.519291 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:49.519324 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260928 22:01:49.657615 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:49.657378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:39645)
0928 22:01:49.657519 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:49.657564 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 22:01:50.520044 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:50.519847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:39649)
0928 22:01:50.519962 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:50.519997 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":43,"threads_started":1}
I20260928 22:01:50.520068 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:50.519812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:41961)
0928 22:01:50.519963 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:50.520004 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260928 22:01:50.658382 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:50.658136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:53547)
0928 22:01:50.658290 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:50.658334 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260928 22:01:51.520747 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:51.520467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:56663)
0928 22:01:51.520614 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:51.520649 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 22:01:51.520771 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:51.520475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:45327)
0928 22:01:51.520615 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:51.520649 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260928 22:01:51.659211 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:51.658942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:59137)
0928 22:01:51.659110 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:51.659156 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260928 22:01:52.521399 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:52.521184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:58263)
0928 22:01:52.521304 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:52.521343 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 22:01:52.521477 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:52.521285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:42285)
0928 22:01:52.521397 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:52.521435 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260928 22:01:52.660035 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:52.659783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:58455)
0928 22:01:52.659941 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:52.659985 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260928 22:01:53.522168 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:53.521911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.67:41137)
0928 22:01:53.522070 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:53.522115 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260928 22:01:53.522168 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:53.521928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.66:45249)
0928 22:01:53.522088 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:53.522115 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260928 22:01:53.661002 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:53.660657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45199 (local address 127.2.165.65:43397)
0928 22:01:53.660904 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:53.660952 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:45199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":146,"threads_started":1}
I20260928 22:01:54.152138 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 17552
I20260928 22:01:54.156986 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 17714
I20260928 22:01:54.161904 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 17851
2026-09-28T22:01:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18281 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:01:54 dist-test-slave-3t36 krb5kdc[18045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:01:54 dist-test-slave-3t36 krb5kdc[18045](info): set up 2 sockets
Sep 28 22:01:54 dist-test-slave-3t36 krb5kdc[18045](info): commencing operation
krb5kdc: starting...
W20260928 22:01:55.224936 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:01:55.352481 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35097
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:35097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:55.435503 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:55.435746 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:55.435772 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:55.437183 18058 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:01:55.437244 18058 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:01:55.437263 18058 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:01:55.437275 18058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:01:55.437288 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:55.437335 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:55.437353 18058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:01:55.437369 18058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:01:55.439085 18058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:35097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:55.439379 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:55.439636 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:55.445288 18064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:55.445288 18066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:55.445390 18058 server_base.cc:1034] running on GCE node
W20260928 22:01:55.445288 18063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:55.445827 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:55.446182 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:55.447317 18058 hybrid_clock.cc:648] HybridClock initialized: now 1790632915447298 us; error 39 us; skew 500 ppm
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:55.450011 18058 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:01:55.450510 18058 webserver.cc:460] Webserver started at http://127.2.165.126:38289/ using document root <none> and password file <none>
I20260928 22:01:55.450740 18058 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:55.450788 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:55.450934 18058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:55.452067 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "a6ff3bff1ec344a3a768503366c0ef4f"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "5ebef2410eba03b34b2d55aebff2f725"
server_key_iv: "13994c6a1b77d7c13fe935ee62eb9d83"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.452448 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "a6ff3bff1ec344a3a768503366c0ef4f"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "5ebef2410eba03b34b2d55aebff2f725"
server_key_iv: "13994c6a1b77d7c13fe935ee62eb9d83"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.453969 18058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:01:55.454875 18073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:55.455101 18058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:55.455183 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "a6ff3bff1ec344a3a768503366c0ef4f"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "5ebef2410eba03b34b2d55aebff2f725"
server_key_iv: "13994c6a1b77d7c13fe935ee62eb9d83"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.455284 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:55.467388 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:55.470592 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:55.470796 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:55.475291 18058 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:35097
I20260928 22:01:55.475322 18125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:35097 every 8 connection(s)
I20260928 22:01:55.475695 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:01:55.476639 18126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:55.477792 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18058
I20260928 22:01:55.477948 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:01:55.478147 2709 external_mini_cluster.cc:1428] Setting key 7494d86b2490299961077f8495d8dd0f
I20260928 22:01:55.479887 18126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Bootstrap starting.
I20260928 22:01:55.480607 18126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:55.480986 18126 log.cc:824] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:55.481822 18126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: No bootstrap required, opened a new log
I20260928 22:01:55.484205 18126 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:01:55.484357 18126 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:55.484380 18126 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ff3bff1ec344a3a768503366c0ef4f, State: Initialized, Role: FOLLOWER
I20260928 22:01:55.484525 18126 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:01:55.484611 18126 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:01:55.484709 18126 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:01:55.484776 18126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:55.485461 18126 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:01:55.485584 18126 leader_election.cc:302] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ff3bff1ec344a3a768503366c0ef4f; no voters:
I20260928 22:01:55.485781 18126 leader_election.cc:288] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:55.485850 18131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:55.486011 18131 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 LEADER]: Becoming Leader. State: Replica: a6ff3bff1ec344a3a768503366c0ef4f, State: Running, Role: LEADER
I20260928 22:01:55.486100 18126 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:01:55.486191 18131 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:01:55.486675 18131 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ff3bff1ec344a3a768503366c0ef4f. Latest consensus state: current_term: 1 leader_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } } }
I20260928 22:01:55.486828 18131 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:55.487092 18140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:01:55.487777 18132 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } } }
I20260928 22:01:55.487934 18140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:01:55.487958 18132 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:01:55.489893 18140 catalog_manager.cc:1347] Generated new cluster ID: eab90900cd114589a304b776f787c1f4
I20260928 22:01:55.489951 18140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:01:55.490834 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:55.478691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.2.165.126:35097)
0928 22:01:55.478902 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:55.478913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:55.478948 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:01:55.479538 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:55.479547 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:55.479614 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:55.479781 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:55.480489 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.481952 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.482701 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.483378 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.483518 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:55.486467 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:55.486471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:55.486472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:55.486485 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:55.488643 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:55.489171 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:55.489174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:55.489174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:55.489300 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:55.489527 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:55.489528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:55.489528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:55.489839 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:55.489987 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:55.490492 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 22:01:55.490629 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260928 22:01:55.523365 18140 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:01:55.524160 18140 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:01:55.529204 18140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Generated new TSK 0
I20260928 22:01:55.529464 18140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 22:01:55.560994 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35097
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:55.647033 18152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:55.647279 18152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:55.647323 18152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:55.648524 18152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:55.648593 18152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:55.648650 18152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:01:55.650206 18152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:55.650478 18152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:55.650743 18152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:55.655998 18158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:55.655980 18157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:55.655980 18160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:55.656193 18152 server_base.cc:1034] running on GCE node
I20260928 22:01:55.656404 18152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:55.656661 18152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:55.657871 18152 hybrid_clock.cc:648] HybridClock initialized: now 1790632915657846 us; error 35 us; skew 500 ppm
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:55.660912 18152 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:01:55.661331 18152 webserver.cc:460] Webserver started at http://127.2.165.65:40191/ using document root <none> and password file <none>
I20260928 22:01:55.661536 18152 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:55.661577 18152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:55.661681 18152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:55.662691 18152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "7b65697fa17548c1b8cace90cbb099ad"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "93f4e520cdb0e778526d7e374212b945"
server_key_iv: "c462f98df167c01ba6b56b9cd37b46d9"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.663009 18152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "7b65697fa17548c1b8cace90cbb099ad"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "93f4e520cdb0e778526d7e374212b945"
server_key_iv: "c462f98df167c01ba6b56b9cd37b46d9"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.664247 18152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:01:55.665091 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:55.665298 18152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:55.665390 18152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "7b65697fa17548c1b8cace90cbb099ad"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "93f4e520cdb0e778526d7e374212b945"
server_key_iv: "c462f98df167c01ba6b56b9cd37b46d9"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.665452 18152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:55.688133 18152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:55.690573 18152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:55.690707 18152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:55.690899 18152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:55.691228 18152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:55.691260 18152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:55.691283 18152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:55.691296 18152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:55.697499 18152 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:35579
I20260928 22:01:55.697566 18280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:35579 every 8 connection(s)
I20260928 22:01:55.697937 18152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:55.706684 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18152
I20260928 22:01:55.706827 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:01:55.707044 2709 external_mini_cluster.cc:1428] Setting key b9decf0ae79acd527847541d6838936f
I20260928 22:01:55.708222 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:55.698745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:58963 (local address 127.2.165.126:35097)
0928 22:01:55.698820 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:55.698823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:55.699011 (+ 188us) server_negotiation.cc:407] Connection header received
0928 22:01:55.699496 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:55.699497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:55.699508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:55.699561 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:55.700523 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.701114 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.702152 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.702376 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.702447 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:55.704994 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:55.704996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:55.704996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:55.705013 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:55.706467 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:55.707005 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:55.707007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:55.707008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:55.707069 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:55.707387 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:55.707388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:55.707389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:55.707535 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:55.707599 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:55.707960 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 22:01:55.708082 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:01:55.708233 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:55.698691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:58963)
0928 22:01:55.698893 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:55.698930 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:55.699414 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:55.699620 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:55.699623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:55.700213 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:55.700430 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:55.700436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.701197 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.701197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:55.702061 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:55.702064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.702484 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.702485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:55.702606 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:55.703216 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:55.703225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:55.704892 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:55.706572 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:55.706580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:55.706583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:55.706926 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:55.707135 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:55.707137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:55.707137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:55.707287 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:55.707605 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:55.707611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:55.707797 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 22:01:55.707913 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:01:55.708050 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":117,"threads_started":1}
I20260928 22:01:55.708918 18281 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35097
I20260928 22:01:55.709033 18281 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:55.709239 18281 heartbeater.cc:511] Master 127.2.165.126:35097 requested a full tablet report, sending...
I20260928 22:01:55.709867 18090 ts_manager.cc:194] Registered new tserver with Master: 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579)
I20260928 22:01:55.710796 18090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:58963
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 22:01:55.735895 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35097
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:55.821910 18286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:55.822170 18286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:55.822211 18286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:55.823421 18286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:55.823489 18286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:55.823535 18286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:01:55.825073 18286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:55.825359 18286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:55.825593 18286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:55.830901 18294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:55.830886 18292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:55.831033 18286 server_base.cc:1034] running on GCE node
W20260928 22:01:55.830946 18291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:55.831291 18286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:55.831550 18286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:55.832787 18286 hybrid_clock.cc:648] HybridClock initialized: now 1790632915832709 us; error 87 us; skew 500 ppm
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:55.835670 18286 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:01:55.836172 18286 webserver.cc:460] Webserver started at http://127.2.165.66:41321/ using document root <none> and password file <none>
I20260928 22:01:55.836374 18286 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:55.836417 18286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:55.836509 18286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:55.837661 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "c40a5ea73c3545ca82cccd746de55147"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "c3bb5ca946a97d0292011034d1eb5858"
server_key_iv: "f8023e2e6cc8eaa1e25e733e04836e9b"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.838011 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "c40a5ea73c3545ca82cccd746de55147"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "c3bb5ca946a97d0292011034d1eb5858"
server_key_iv: "f8023e2e6cc8eaa1e25e733e04836e9b"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.839373 18286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:01:55.840232 18301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:55.840420 18286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:55.840507 18286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "c40a5ea73c3545ca82cccd746de55147"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "c3bb5ca946a97d0292011034d1eb5858"
server_key_iv: "f8023e2e6cc8eaa1e25e733e04836e9b"
server_key_version: "encryptionkey@0"
I20260928 22:01:55.840570 18286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:55.860927 18286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:55.863741 18286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:55.863881 18286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:55.864084 18286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:55.864372 18286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:55.864404 18286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:55.864427 18286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:55.864439 18286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:55.870740 18286 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:43427
I20260928 22:01:55.870824 18414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:43427 every 8 connection(s)
I20260928 22:01:55.871156 18286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:01:55.871363 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18286
I20260928 22:01:55.871495 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:01:55.871726 2709 external_mini_cluster.cc:1428] Setting key e99176836c835728b82b3a1efbc17272
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 22:01:55.882757 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:55.872124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:57677 (local address 127.2.165.126:35097)
0928 22:01:55.872175 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:55.872177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:55.872484 (+ 307us) server_negotiation.cc:407] Connection header received
0928 22:01:55.873693 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:55.873695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:55.873709 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:55.873777 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:55.874718 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.875286 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.876392 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.876596 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.876708 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:55.879313 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:55.879315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:55.879315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:55.879324 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:55.881087 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:55.881551 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:55.881552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:55.881553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:55.881617 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:55.881893 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:55.881894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:55.881894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:55.882054 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:55.882203 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:55.882514 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 22:01:55.882661 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
I20260928 22:01:55.882992 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:55.872154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:57677)
0928 22:01:55.872334 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:55.872386 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:55.873548 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:55.873866 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:55.873871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:55.874272 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:55.874596 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:55.874604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.875429 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.875431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:55.876271 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:55.876273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:55.876715 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:55.876717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:55.876856 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:55.877442 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:55.877452 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:55.879206 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:55.881184 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:55.881221 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:55.881223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:55.881472 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:55.881699 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:55.881700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:55.881700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:55.881819 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:55.882188 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:55.882196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:55.882360 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 22:01:55.882489 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 22:01:55.882671 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 22:01:55.883490 18415 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35097
I20260928 22:01:55.883611 18415 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:55.883839 18415 heartbeater.cc:511] Master 127.2.165.126:35097 requested a full tablet report, sending...
I20260928 22:01:55.884279 18090 ts_manager.cc:194] Registered new tserver with Master: c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
I20260928 22:01:55.884886 18090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:57677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 22:01:55.901006 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35097
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:01:55.985548 18420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:01:55.985831 18420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:01:55.985860 18420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:01:55.987198 18420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:01:55.987247 18420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:01:55.987303 18420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:01:55.988906 18420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:01:55.989108 18420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:01:55.989306 18420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:01:55.994493 18425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:55.994493 18428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:01:55.994503 18426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:01:55.995018 18420 server_base.cc:1034] running on GCE node
I20260928 22:01:55.995237 18420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:01:55.995453 18420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:01:55.996598 18420 hybrid_clock.cc:648] HybridClock initialized: now 1790632915996577 us; error 43 us; skew 500 ppm
Sep 28 22:01:55 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:01:55.999298 18420 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:01:55.999837 18420 webserver.cc:460] Webserver started at http://127.2.165.67:33609/ using document root <none> and password file <none>
I20260928 22:01:56.000103 18420 fs_manager.cc:359] Metadata directory not provided
I20260928 22:01:56.000151 18420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:01:56.000278 18420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:01:56.001466 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "a4f27afc473543dc9201dce918e1abf9"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "8085888e2149e670e306d05eecccdcb7"
server_key_iv: "6e79b735911373e61073d021031323e4"
server_key_version: "encryptionkey@0"
I20260928 22:01:56.001834 18420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "a4f27afc473543dc9201dce918e1abf9"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "8085888e2149e670e306d05eecccdcb7"
server_key_iv: "6e79b735911373e61073d021031323e4"
server_key_version: "encryptionkey@0"
I20260928 22:01:56.003254 18420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:01:56.004101 18435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:56.004289 18420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:01:56.004379 18420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "a4f27afc473543dc9201dce918e1abf9"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "8085888e2149e670e306d05eecccdcb7"
server_key_iv: "6e79b735911373e61073d021031323e4"
server_key_version: "encryptionkey@0"
I20260928 22:01:56.004456 18420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:01:56.027618 18420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:01:56.030189 18420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:01:56.030313 18420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:01:56.030541 18420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:01:56.030910 18420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:01:56.030946 18420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:56.030987 18420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:01:56.031013 18420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:01:56.037155 18420 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:40821
I20260928 22:01:56.037230 18548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:40821 every 8 connection(s)
I20260928 22:01:56.037539 18420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:01:56 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:01:56.046682 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18420
I20260928 22:01:56.046875 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:01:56.047079 2709 external_mini_cluster.cc:1428] Setting key aaafa2a40b63cc5ac92cfa74c6e6f69d
I20260928 22:01:56.048446 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.038387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:55459 (local address 127.2.165.126:35097)
0928 22:01:56.038481 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.038484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.038627 (+ 143us) server_negotiation.cc:407] Connection header received
0928 22:01:56.039275 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.039276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.039287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.039351 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:56.040423 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.041053 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.042236 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.042432 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.042497 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.045113 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:56.045115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:56.045115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:56.045123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:56.046661 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:56.047181 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:56.047183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:56.047183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:56.047257 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:56.047618 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:56.047619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:56.047619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:56.047787 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:56.047847 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.048280 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.048370 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 22:01:56.048645 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.038305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:55459)
0928 22:01:56.038494 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.038536 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:56.039117 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:56.039437 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:56.039441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:56.040112 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:01:56.040329 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.040334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.041141 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.041143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.042137 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.042139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.042538 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.042540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.042661 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.043291 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:01:56.043300 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:01:56.045017 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:01:56.046753 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:56.046762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:56.046765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:56.047093 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:56.047376 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:01:56.047377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:01:56.047377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:01:56.047547 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:01:56.047924 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:01:56.047932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:01:56.048099 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 22:01:56.048237 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 22:01:56.048428 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260928 22:01:56.049176 18549 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35097
I20260928 22:01:56.049324 18549 heartbeater.cc:464] Registering TS with master...
I20260928 22:01:56.049531 18549 heartbeater.cc:511] Master 127.2.165.126:35097 requested a full tablet report, sending...
I20260928 22:01:56.049925 18090 ts_manager.cc:194] Registered new tserver with Master: a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821)
I20260928 22:01:56.050344 18090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:55459
I20260928 22:01:56.059732 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:01:56.068063 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.061358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.2.165.126:35097)
0928 22:01:56.061415 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.061416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.061447 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:01:56.061532 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.061533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.061542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.061598 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:01:56.062221 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.062765 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.063541 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.063728 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.063788 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.065062 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:01:56.065063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:01:56.065063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:01:56.065070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:01:56.066590 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:56.067045 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:56.067046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:56.067046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:56.067106 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:01:56.067390 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:01:56.067391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:01:56.067391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:01:56.067568 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:01:56.067663 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.067911 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.067987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:01:56.070019 18090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:01:56.071730 18090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:01:56.078706 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.074646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.2.165.65:35579)
0928 22:01:56.074840 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.074846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.074857 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:01:56.074901 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.074904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.074922 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.075011 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:56.075295 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.076133 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.077557 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.078202 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.078251 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.078320 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.078573 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.078614 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260928 22:01:56.078868 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.074410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:40821 (local address 127.0.0.1:40024)
0928 22:01:56.074795 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.074804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:56.074843 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:56.075189 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:56.075189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:56.075191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:56.075220 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.075221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.076324 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.076325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.077838 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.077841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.078506 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.078507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.078574 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.078578 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:01:56.078628 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:01:56.078690 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":47,"thread_start_us":57,"threads_started":1}
I20260928 22:01:56.078976 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.074410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35579 (local address 127.0.0.1:34174)
0928 22:01:56.074585 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.074630 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:56.074692 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:56.075103 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:56.075106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:56.075108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:56.075203 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.075207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.076269 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.076270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.077459 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.077461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.078381 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.078382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.078461 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.078473 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 22:01:56.078837 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 22:01:56.078891 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260928 22:01:56.078773 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.074869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.2.165.67:40821)
0928 22:01:56.075026 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.075031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.075042 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:01:56.075083 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.075084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.075097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.075185 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:56.075328 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.076175 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.077927 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.078411 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.078494 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.078548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.078640 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.078688 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260928 22:01:56.079780 18483 tablet_service.cc:1467] Processing CreateTablet for tablet 610421753db34b26860cda32dafbc1ed (DEFAULT_TABLE table=test-table [id=da7c37d87b8742cd8e97e2ae4cb4ca5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:56.079847 18215 tablet_service.cc:1467] Processing CreateTablet for tablet 610421753db34b26860cda32dafbc1ed (DEFAULT_TABLE table=test-table [id=da7c37d87b8742cd8e97e2ae4cb4ca5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:56.080164 18215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610421753db34b26860cda32dafbc1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:56.080164 18483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610421753db34b26860cda32dafbc1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:56.080209 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.074410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:43427 (local address 127.0.0.1:51590)
0928 22:01:56.074911 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.074919 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:56.074956 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:56.076496 (+ 1540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:56.076497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:56.076500 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:56.076601 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.076603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.077882 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.077883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.078924 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.078927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.079976 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.079977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.080039 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.080042 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:01:56.080098 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:01:56.080137 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":48,"thread_start_us":122,"threads_started":1}
I20260928 22:01:56.080432 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.074703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.2.165.66:43427)
0928 22:01:56.076333 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.076339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.076350 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:01:56.076388 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.076391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.076406 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.076488 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:56.076719 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.077782 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.079018 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.079827 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.079889 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.079942 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.080112 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.080149 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1587,"thread_start_us":75,"threads_started":1}
I20260928 22:01:56.081030 18349 tablet_service.cc:1467] Processing CreateTablet for tablet 610421753db34b26860cda32dafbc1ed (DEFAULT_TABLE table=test-table [id=da7c37d87b8742cd8e97e2ae4cb4ca5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:01:56.081341 18349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610421753db34b26860cda32dafbc1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:01:56.083642 18567 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap starting.
I20260928 22:01:56.083887 18566 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Bootstrap starting.
I20260928 22:01:56.084340 18568 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Bootstrap starting.
I20260928 22:01:56.084604 18566 tablet_bootstrap.cc:654] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:56.085049 18568 tablet_bootstrap.cc:654] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:56.085175 18566 log.cc:824] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:56.085750 18568 log.cc:824] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:56.086120 18566 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: No bootstrap required, opened a new log
I20260928 22:01:56.086207 18567 tablet_bootstrap.cc:654] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Neither blocks nor log segments found. Creating new log.
I20260928 22:01:56.086205 18566 ts_tablet_manager.cc:1397] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:01:56.086658 18568 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: No bootstrap required, opened a new log
I20260928 22:01:56.086741 18568 ts_tablet_manager.cc:1397] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:01:56.086829 18567 log.cc:824] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Log is configured to *not* fsync() on all Append() calls
I20260928 22:01:56.087687 18567 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: No bootstrap required, opened a new log
I20260928 22:01:56.087788 18567 ts_tablet_manager.cc:1397] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 22:01:56.088537 18566 raft_consensus.cc:348] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.088793 18566 raft_consensus.cc:374] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:56.088636 18568 raft_consensus.cc:348] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.088843 18566 raft_consensus.cc:729] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f27afc473543dc9201dce918e1abf9, State: Initialized, Role: FOLLOWER
I20260928 22:01:56.088881 18568 raft_consensus.cc:374] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:56.088933 18568 raft_consensus.cc:729] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40a5ea73c3545ca82cccd746de55147, State: Initialized, Role: FOLLOWER
I20260928 22:01:56.088984 18566 consensus_queue.cc:260] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.089061 18568 consensus_queue.cc:260] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.089273 18566 ts_tablet_manager.cc:1428] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:01:56.089403 18549 heartbeater.cc:503] Master 127.2.165.126:35097 was elected leader, sending a full tablet report...
I20260928 22:01:56.089438 18568 ts_tablet_manager.cc:1428] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 22:01:56.089473 18415 heartbeater.cc:503] Master 127.2.165.126:35097 was elected leader, sending a full tablet report...
I20260928 22:01:56.090194 18567 raft_consensus.cc:348] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.090386 18567 raft_consensus.cc:374] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:01:56.090430 18567 raft_consensus.cc:729] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b65697fa17548c1b8cace90cbb099ad, State: Initialized, Role: FOLLOWER
I20260928 22:01:56.090597 18567 consensus_queue.cc:260] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.090909 18567 ts_tablet_manager.cc:1428] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:01:56.090906 18281 heartbeater.cc:503] Master 127.2.165.126:35097 was elected leader, sending a full tablet report...
W20260928 22:01:56.121948 18416 tablet.cc:2369] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:01:56.198738 18282 tablet.cc:2369] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:01:56.199566 18572 raft_consensus.cc:482] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:01:56.199698 18572 raft_consensus.cc:504] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.200132 18572 leader_election.cc:288] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579), c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
I20260928 22:01:56.204352 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.200427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:33739 (local address 127.2.165.65:35579)
0928 22:01:56.200472 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.200474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.200591 (+ 117us) server_negotiation.cc:407] Connection header received
0928 22:01:56.200641 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.200642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.200651 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.200739 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:56.200912 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.201690 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.203398 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.203983 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.204065 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.204074 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.204236 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.204278 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:01:56.204352 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.200392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:52001 (local address 127.2.165.66:43427)
0928 22:01:56.200460 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.200462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.200522 (+ 60us) server_negotiation.cc:407] Connection header received
0928 22:01:56.200611 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.200613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.200624 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.200788 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:01:56.200962 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.201691 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.203394 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.203983 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.204065 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.204074 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.204235 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.204278 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:01:56.204377 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.200318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:43427 (local address 127.2.165.67:52001)
0928 22:01:56.200453 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.200465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:56.200529 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:56.200857 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:56.200858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:56.200859 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:56.200909 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.200910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.201824 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.201825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.203270 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.203274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.204070 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.204071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.204157 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.204163 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:01:56.204221 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:01:56.204319 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":82}
I20260928 22:01:56.204382 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.200326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35579 (local address 127.2.165.67:33739)
0928 22:01:56.200527 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.200536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:01:56.200585 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:01:56.200744 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:01:56.200745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:01:56.200747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:01:56.200824 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.200826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.201792 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.201794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.203270 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:01:56.203274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.204070 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.204071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:01:56.204157 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.204164 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:01:56.204221 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:01:56.204288 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":118,"thread_start_us":97,"threads_started":1}
I20260928 22:01:56.204902 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610421753db34b26860cda32dafbc1ed" candidate_uuid: "a4f27afc473543dc9201dce918e1abf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40a5ea73c3545ca82cccd746de55147" is_pre_election: true
I20260928 22:01:56.205037 18369 raft_consensus.cc:2454] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f27afc473543dc9201dce918e1abf9 in term 0.
I20260928 22:01:56.205255 18439 leader_election.cc:302] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4f27afc473543dc9201dce918e1abf9, c40a5ea73c3545ca82cccd746de55147; no voters:
I20260928 22:01:56.205260 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610421753db34b26860cda32dafbc1ed" candidate_uuid: "a4f27afc473543dc9201dce918e1abf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b65697fa17548c1b8cace90cbb099ad" is_pre_election: true
I20260928 22:01:56.205408 18235 raft_consensus.cc:2454] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f27afc473543dc9201dce918e1abf9 in term 0.
I20260928 22:01:56.205415 18572 raft_consensus.cc:2781] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:01:56.205464 18572 raft_consensus.cc:482] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:01:56.205500 18572 raft_consensus.cc:3037] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:56.206243 18572 raft_consensus.cc:504] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.206422 18572 leader_election.cc:288] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579), c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
I20260928 22:01:56.206563 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610421753db34b26860cda32dafbc1ed" candidate_uuid: "a4f27afc473543dc9201dce918e1abf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b65697fa17548c1b8cace90cbb099ad"
I20260928 22:01:56.206576 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610421753db34b26860cda32dafbc1ed" candidate_uuid: "a4f27afc473543dc9201dce918e1abf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40a5ea73c3545ca82cccd746de55147"
I20260928 22:01:56.206653 18235 raft_consensus.cc:3037] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:56.206656 18369 raft_consensus.cc:3037] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:01:56.207620 18369 raft_consensus.cc:2454] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f27afc473543dc9201dce918e1abf9 in term 1.
I20260928 22:01:56.207620 18235 raft_consensus.cc:2454] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f27afc473543dc9201dce918e1abf9 in term 1.
I20260928 22:01:56.207837 18436 leader_election.cc:302] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad, a4f27afc473543dc9201dce918e1abf9; no voters:
I20260928 22:01:56.207939 18572 raft_consensus.cc:2781] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:01:56.208097 18572 raft_consensus.cc:686] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 1 LEADER]: Becoming Leader. State: Replica: a4f27afc473543dc9201dce918e1abf9, State: Running, Role: LEADER
I20260928 22:01:56.208173 18572 consensus_queue.cc:237] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:01:56.209173 18089 catalog_manager.cc:5495] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4f27afc473543dc9201dce918e1abf9 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "a4f27afc473543dc9201dce918e1abf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } health_report { overall_health: UNKNOWN } } }
I20260928 22:01:56.224236 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.221757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.2.165.67:40821)
0928 22:01:56.221821 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:01:56.221823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:01:56.221836 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:01:56.221881 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:01:56.221882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:01:56.221904 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:01:56.221989 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:01:56.222124 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.222607 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.223385 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:01:56.223603 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:01:56.223685 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:01:56.223846 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:01:56.223966 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:01:56.224051 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:01:56.224102 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 22:01:56.224124 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:01:56.226266 18235 raft_consensus.cc:1261] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Refusing update from remote peer a4f27afc473543dc9201dce918e1abf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:56.226305 18369 raft_consensus.cc:1261] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Refusing update from remote peer a4f27afc473543dc9201dce918e1abf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:01:56.226675 18572 consensus_queue.cc:1035] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:56.226835 18576 consensus_queue.cc:1035] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:01:56.229349 18579 mvcc.cc:205] Tried to move back new op lower bound from 7334432424859484160 to 7334432424789655552. Current Snapshot: MvccSnapshot[applied={T|T < 7334432424859484160}]
I20260928 22:01:56.230528 18580 mvcc.cc:205] Tried to move back new op lower bound from 7334432424859484160 to 7334432424789655552. Current Snapshot: MvccSnapshot[applied={T|T < 7334432424859484160 or (T in {7334432424859484160})}]
I20260928 22:01:56.232133 18090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:56.232234 18090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:01:56.233002 18090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Sending DeleteTablet for 3 replicas of tablet 610421753db34b26860cda32dafbc1ed
I20260928 22:01:56.233321 18483 tablet_service.cc:1514] Processing DeleteTablet for tablet 610421753db34b26860cda32dafbc1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:40024
I20260928 22:01:56.233353 18215 tablet_service.cc:1514] Processing DeleteTablet for tablet 610421753db34b26860cda32dafbc1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:34174
I20260928 22:01:56.233403 18349 tablet_service.cc:1514] Processing DeleteTablet for tablet 610421753db34b26860cda32dafbc1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:51590
I20260928 22:01:56.233470 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18152
I20260928 22:01:56.234300 18589 tablet_replica.cc:330] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: stopping tablet replica
I20260928 22:01:56.234431 18589 raft_consensus.cc:2229] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:01:56.234565 18589 raft_consensus.cc:2258] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:01:56.234989 18589 ts_tablet_manager.cc:1905] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:01:56.236589 18589 ts_tablet_manager.cc:1918] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:01:56.236730 18589 log.cc:1196] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/610421753db34b26860cda32dafbc1ed
I20260928 22:01:56.237005 18589 ts_tablet_manager.cc:1939] T 610421753db34b26860cda32dafbc1ed P a4f27afc473543dc9201dce918e1abf9: Deleting consensus metadata
I20260928 22:01:56.237406 18075 catalog_manager.cc:4841] TS a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821): tablet 610421753db34b26860cda32dafbc1ed (table test-table [id=da7c37d87b8742cd8e97e2ae4cb4ca5d]) successfully deleted
W20260928 22:01:56.239190 18074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:35579 (error 108)
I20260928 22:01:56.239430 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18286
I20260928 22:01:56.240065 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.239625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35579 (local address 127.0.0.1:34182)
0928 22:01:56.239984 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.240023 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:35579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
W20260928 22:01:56.240191 18074 catalog_manager.cc:4584] TS 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 610421753db34b26860cda32dafbc1ed: Network error: Client connection negotiation failed: client connection to 127.2.165.65:35579: connect: Connection refused (error 111)
I20260928 22:01:56.248045 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18420
I20260928 22:01:56.248184 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:01:56.248040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:43427 (local address 127.0.0.1:51592)
0928 22:01:56.248104 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:01:56.248139 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.66:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:01:56.252987 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18058
I20260928 22:02:16.259034 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35097
--webserver_interface=127.2.165.126
--webserver_port=38289
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:35097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:16.360275 18593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:16.360507 18593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:16.360532 18593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:16.361889 18593 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:02:16.361970 18593 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:02:16.361984 18593 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:02:16.361996 18593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:02:16.362010 18593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:16.362022 18593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:16.362073 18593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:02:16.362098 18593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:02:16.363672 18593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:35097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:16.363953 18593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:16.364215 18593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:16.375437 18599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:16.375439 18602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:16.375465 18593 server_base.cc:1034] running on GCE node
W20260928 22:02:16.375439 18600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:16.376039 18593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:16.376451 18593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:16.377650 18593 hybrid_clock.cc:648] HybridClock initialized: now 1790632936377629 us; error 35 us; skew 500 ppm
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:16.386508 18593 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:02:16.387077 18593 webserver.cc:460] Webserver started at http://127.2.165.126:38289/ using document root <none> and password file <none>
I20260928 22:02:16.387342 18593 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:16.387441 18593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:16.389034 18593 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:02:16.389822 18609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:16.390045 18593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:16.390170 18593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "a6ff3bff1ec344a3a768503366c0ef4f"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "5ebef2410eba03b34b2d55aebff2f725"
server_key_iv: "13994c6a1b77d7c13fe935ee62eb9d83"
server_key_version: "encryptionkey@0"
I20260928 22:02:16.390600 18593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:16.403023 18593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:16.405668 18593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:16.405843 18593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:16.410187 18593 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:35097
I20260928 22:02:16.410223 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:35097 every 8 connection(s)
I20260928 22:02:16.410643 18593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:02:16.412271 18662 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:02:16.412993 18662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Bootstrap starting.
I20260928 22:02:16.416378 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18593
I20260928 22:02:16.416805 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:35579
--local_ip_for_outbound_sockets=127.2.165.65
--tserver_master_addrs=127.2.165.126:35097
--webserver_port=40191
--webserver_interface=127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:02:16.421686 18662 log.cc:824] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:16.423704 18662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:02:16.424041 18662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Bootstrap complete.
I20260928 22:02:16.426370 18662 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:16.426628 18662 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ff3bff1ec344a3a768503366c0ef4f, State: Initialized, Role: FOLLOWER
I20260928 22:02:16.426821 18662 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:16.426910 18662 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:02:16.426971 18662 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:02:16.427031 18662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:02:16.428126 18662 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:16.428256 18662 leader_election.cc:302] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ff3bff1ec344a3a768503366c0ef4f; no voters:
I20260928 22:02:16.428485 18662 leader_election.cc:288] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:02:16.428532 18666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:02:16.428660 18666 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 LEADER]: Becoming Leader. State: Replica: a6ff3bff1ec344a3a768503366c0ef4f, State: Running, Role: LEADER
I20260928 22:02:16.428788 18662 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:02:16.428774 18666 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:16.429023 18666 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ff3bff1ec344a3a768503366c0ef4f. Latest consensus state: current_term: 2 leader_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } } }
I20260928 22:02:16.429087 18666 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:16.429297 18667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } } }
I20260928 22:02:16.429390 18667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:16.429805 18670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:02:16.430814 18670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:02:16.431021 18670 catalog_manager.cc:1259] Loaded cluster ID: eab90900cd114589a304b776f787c1f4
I20260928 22:02:16.431200 18670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:02:16.432667 18670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:02:16.432920 18670 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Loaded TSK: 0
I20260928 22:02:16.445715 18670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Generated new TSK 1
I20260928 22:02:16.445856 18670 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:02:16.511783 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:16.512207 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:16.512241 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:16.513652 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:16.513706 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:16.513739 18664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:02:16.515348 18664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:35579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=40191
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:16.515621 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:16.515901 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:16.521109 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:16.521092 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:16.521070 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:16.521824 18664 server_base.cc:1034] running on GCE node
I20260928 22:02:16.522094 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:16.522348 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:16.523557 18664 hybrid_clock.cc:648] HybridClock initialized: now 1790632936523533 us; error 40 us; skew 500 ppm
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:16.526566 18664 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:02:16.527117 18664 webserver.cc:460] Webserver started at http://127.2.165.65:40191/ using document root <none> and password file <none>
I20260928 22:02:16.527381 18664 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:16.527475 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:16.529026 18664 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:02:16.529814 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:16.530036 18664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:16.530138 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "7b65697fa17548c1b8cace90cbb099ad"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "93f4e520cdb0e778526d7e374212b945"
server_key_iv: "c462f98df167c01ba6b56b9cd37b46d9"
server_key_version: "encryptionkey@0"
I20260928 22:02:16.530530 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:16.562012 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:16.564586 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:16.564823 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:16.565099 18664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:16.565761 18706 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:02:16.567404 18664 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:02:16.567494 18664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:02:16.567536 18664 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:02:16.568300 18664 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:02:16.568359 18664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:16.568439 18706 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap starting.
I20260928 22:02:16.575871 18664 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:35579
I20260928 22:02:16.575971 18813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:35579 every 8 connection(s)
I20260928 22:02:16.576323 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:02:16.578245 18706 log.cc:824] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:16.580395 18706 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:02:16.580783 18706 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap complete.
I20260928 22:02:16.581730 18706 ts_tablet_manager.cc:1397] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260928 22:02:16.583021 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18664
I20260928 22:02:16.582892 18706 raft_consensus.cc:348] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.583280 18706 raft_consensus.cc:729] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b65697fa17548c1b8cace90cbb099ad, State: Initialized, Role: FOLLOWER
I20260928 22:02:16.583348 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:43427
--local_ip_for_outbound_sockets=127.2.165.66
--tserver_master_addrs=127.2.165.126:35097
--webserver_port=41321
--webserver_interface=127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:02:16.583463 18706 consensus_queue.cc:260] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.583784 18706 ts_tablet_manager.cc:1428] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:16.588406 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.577135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:45437)
0928 22:02:16.577356 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:16.577436 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:16.578128 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:16.578675 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:16.578680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:16.578971 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:16.579162 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.579168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.580384 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.580385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.581396 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.581400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.582116 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.582118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.582263 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.582849 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:16.582858 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:16.584498 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:16.586628 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.586638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.586640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.586885 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.587182 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.587183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.587184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.587296 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.587710 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:16.587717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:16.587860 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 22:02:16.587974 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 22:02:16.588186 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":105,"threads_started":1}
I20260928 22:02:16.588406 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.577307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:45437 (local address 127.2.165.126:35097)
0928 22:02:16.577491 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.577501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.577543 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:02:16.578403 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.578411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.578484 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.578642 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:16.579285 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.580252 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.581510 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.581991 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.582125 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.584624 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:16.584629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:16.584629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:16.584643 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:16.586492 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.586979 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.586982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.586983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.587101 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.587366 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.587367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.587367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.587630 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:16.587732 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.588105 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.588247 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260928 22:02:16.589447 18814 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35097
I20260928 22:02:16.589586 18814 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:16.589773 18814 heartbeater.cc:511] Master 127.2.165.126:35097 requested a full tablet report, sending...
I20260928 22:02:16.590567 18626 ts_manager.cc:194] Registered new tserver with Master: 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579)
I20260928 22:02:16.591621 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:45437
I20260928 22:02:16.592664 18814 heartbeater.cc:503] Master 127.2.165.126:35097 was elected leader, sending a full tablet report...
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.65@KRBTEST.COM
I20260928 22:02:16.601162 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.591221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35579 (local address 127.0.0.1:57434)
0928 22:02:16.591471 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:16.591510 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:16.591587 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:16.592181 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:16.592187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:16.592527 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:16.592591 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.592594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.594040 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.594041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.594907 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.594911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.595562 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.595563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.595625 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.596225 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:16.596229 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:16.597635 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:16.599770 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.599777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.599778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.600050 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.600272 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.600273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.600273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.600332 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.600740 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:16.600747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:16.600802 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:02:16.600907 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 22:02:16.601071 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260928 22:02:16.601194 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.591305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.2.165.65:35579)
0928 22:02:16.591831 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.591836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.591849 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:02:16.592001 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.592011 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.592030 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.592182 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:16.592724 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.593921 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.595003 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.595477 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.595551 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.597722 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:16.597731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:16.597732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:16.597738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:16.599675 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.600137 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.600140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.600141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.600206 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.600394 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.600395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.600395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.600607 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:16.600739 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.600973 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.601089 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":92,"threads_started":1}
I20260928 22:02:16.601635 18748 tablet_service.cc:1514] Processing DeleteTablet for tablet 610421753db34b26860cda32dafbc1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:57434
I20260928 22:02:16.602221 18823 tablet_replica.cc:330] stopping tablet replica
I20260928 22:02:16.602357 18823 raft_consensus.cc:2229] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:16.602461 18823 raft_consensus.cc:2258] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:16.602779 18823 ts_tablet_manager.cc:1905] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:16.605111 18823 ts_tablet_manager.cc:1918] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:16.605225 18823 log.cc:1196] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal/wals/610421753db34b26860cda32dafbc1ed
I20260928 22:02:16.605496 18823 ts_tablet_manager.cc:1939] T 610421753db34b26860cda32dafbc1ed P 7b65697fa17548c1b8cace90cbb099ad: Deleting consensus metadata
I20260928 22:02:16.606078 18610 catalog_manager.cc:4841] TS 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579): tablet 610421753db34b26860cda32dafbc1ed (table test-table [id=da7c37d87b8742cd8e97e2ae4cb4ca5d]) successfully deleted
W20260928 22:02:16.672139 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:16.672385 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:16.672428 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:16.673709 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:16.673784 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:16.673830 18820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:02:16.675390 18820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:43427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=41321
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:16.675675 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:16.675915 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:16.681277 18829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:16.681334 18828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:16.681339 18831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:16.681399 18820 server_base.cc:1034] running on GCE node
I20260928 22:02:16.681736 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:16.682003 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:16.683190 18820 hybrid_clock.cc:648] HybridClock initialized: now 1790632936683173 us; error 40 us; skew 500 ppm
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:16.685935 18820 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:02:16.686465 18820 webserver.cc:460] Webserver started at http://127.2.165.66:41321/ using document root <none> and password file <none>
I20260928 22:02:16.686707 18820 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:16.686789 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:16.688277 18820 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:02:16.689045 18838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:16.689241 18820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:16.689352 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "c40a5ea73c3545ca82cccd746de55147"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "c3bb5ca946a97d0292011034d1eb5858"
server_key_iv: "f8023e2e6cc8eaa1e25e733e04836e9b"
server_key_version: "encryptionkey@0"
I20260928 22:02:16.689759 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:16.706470 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:16.708781 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:16.708977 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:16.709239 18820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:16.709707 18845 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:02:16.711239 18820 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:02:16.711314 18820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:02:16.711359 18820 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:02:16.712038 18820 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:02:16.712106 18820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:16.712188 18845 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Bootstrap starting.
I20260928 22:02:16.719450 18820 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:43427
I20260928 22:02:16.719558 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:43427 every 8 connection(s)
I20260928 22:02:16.719892 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:02:16.721931 18845 log.cc:824] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:16.724406 18845 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:02:16.724787 18845 tablet_bootstrap.cc:492] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Bootstrap complete.
I20260928 22:02:16.725931 18845 ts_tablet_manager.cc:1397] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:16.727188 18845 raft_consensus.cc:348] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.727540 18845 raft_consensus.cc:729] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40a5ea73c3545ca82cccd746de55147, State: Initialized, Role: FOLLOWER
I20260928 22:02:16.727696 18845 consensus_queue.cc:260] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.728046 18845 ts_tablet_manager.cc:1428] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 22:02:16.729251 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18820
I20260928 22:02:16.729621 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:40821
--local_ip_for_outbound_sockets=127.2.165.67
--tserver_master_addrs=127.2.165.126:35097
--webserver_port=33609
--webserver_interface=127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:02:16.730985 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.720733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:57999 (local address 127.2.165.126:35097)
0928 22:02:16.720809 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.720811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.721097 (+ 286us) server_negotiation.cc:407] Connection header received
0928 22:02:16.721816 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.721817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.721832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.721886 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:16.722701 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.723207 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.724731 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.725008 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.725101 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.727606 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:16.727608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:16.727608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:16.727618 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:16.729324 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.729882 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.729883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.729884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.729953 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.730235 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.730236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.730236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.730370 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:16.730446 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.730785 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.730893 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:02:16.731172 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.720662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:57999)
0928 22:02:16.720918 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:16.720986 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:16.721702 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:16.721965 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:16.721969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:16.722304 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:16.722583 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.722590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.723326 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.723327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.724595 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.724598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.725109 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.725111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.725268 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.725833 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:16.725845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:16.727483 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:16.729432 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.729438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.729440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.729813 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.730024 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.730025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.730025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.730154 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.730437 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:16.730444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:16.730622 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 22:02:16.730761 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 22:02:16.730940 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260928 22:02:16.731742 18953 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35097
I20260928 22:02:16.731865 18953 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:16.732062 18953 heartbeater.cc:511] Master 127.2.165.126:35097 requested a full tablet report, sending...
I20260928 22:02:16.732661 18626 ts_manager.cc:194] Registered new tserver with Master: c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
I20260928 22:02:16.733268 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:57999
I20260928 22:02:16.734174 18953 heartbeater.cc:503] Master 127.2.165.126:35097 was elected leader, sending a full tablet report...
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.66@KRBTEST.COM
I20260928 22:02:16.741740 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.733059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:43427 (local address 127.0.0.1:41496)
0928 22:02:16.733112 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:16.733130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:16.733167 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:16.733703 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:16.733704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:16.733841 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:16.733884 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.733886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.735059 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.735060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.735608 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.735610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.736310 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.736311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.736359 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.736881 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:16.736887 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:16.738120 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:16.740444 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.740445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.740446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.740763 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.741020 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.741020 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.741021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.741082 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.741467 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:16.741467 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:16.741510 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:02:16.741562 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 22:02:16.741666 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:02:16.742007 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.733250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.2.165.66:43427)
0928 22:02:16.733558 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.733562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.733573 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:02:16.733609 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.733612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.733628 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.733697 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:16.733963 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.734974 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.735692 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.736228 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.736310 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.738217 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:16.738226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:16.738227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:16.738235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:16.740329 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.740862 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.740867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.740867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.740942 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.741158 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.741158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.741159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.741392 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:16.741494 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.741605 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.741724 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260928 22:02:16.742399 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 610421753db34b26860cda32dafbc1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:01:56 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:41496
I20260928 22:02:16.742928 18960 tablet_replica.cc:330] stopping tablet replica
I20260928 22:02:16.743040 18960 raft_consensus.cc:2229] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:16.743109 18960 raft_consensus.cc:2258] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:16.743436 18960 ts_tablet_manager.cc:1905] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:16.746017 18960 ts_tablet_manager.cc:1918] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:16.746100 18960 log.cc:1196] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/610421753db34b26860cda32dafbc1ed
I20260928 22:02:16.746330 18960 ts_tablet_manager.cc:1939] T 610421753db34b26860cda32dafbc1ed P c40a5ea73c3545ca82cccd746de55147: Deleting consensus metadata
I20260928 22:02:16.746886 18613 catalog_manager.cc:4841] TS c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427): tablet 610421753db34b26860cda32dafbc1ed (table test-table [id=da7c37d87b8742cd8e97e2ae4cb4ca5d]) successfully deleted
W20260928 22:02:16.816984 18958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:16.817194 18958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:16.817217 18958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:16.818308 18958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:16.818344 18958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:16.818372 18958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:02:16.819769 18958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:40821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=33609
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:16.819958 18958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:16.820140 18958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:16.825436 18965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:16.825418 18966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:16.825423 18968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:16.825479 18958 server_base.cc:1034] running on GCE node
I20260928 22:02:16.825866 18958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:16.826126 18958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:16.827265 18958 hybrid_clock.cc:648] HybridClock initialized: now 1790632936827260 us; error 38 us; skew 500 ppm
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:16.830195 18958 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:02:16.830585 18958 webserver.cc:460] Webserver started at http://127.2.165.67:33609/ using document root <none> and password file <none>
I20260928 22:02:16.830770 18958 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:16.830822 18958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:16.832170 18958 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:16.832863 18975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:16.833034 18958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:16.833118 18958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "a4f27afc473543dc9201dce918e1abf9"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "8085888e2149e670e306d05eecccdcb7"
server_key_iv: "6e79b735911373e61073d021031323e4"
server_key_version: "encryptionkey@0"
I20260928 22:02:16.833448 18958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:16.850399 18958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:16.852540 18958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:16.852756 18958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:16.852972 18958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:16.853317 18958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:16.853353 18958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:16.853375 18958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:16.853422 18958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:16.859352 18958 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:40821
I20260928 22:02:16.859393 19088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:40821 every 8 connection(s)
I20260928 22:02:16.859763 18958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:02:16 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:16.867096 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 18958
I20260928 22:02:16.869920 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.860619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:36421 (local address 127.2.165.126:35097)
0928 22:02:16.860773 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.860776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.861016 (+ 240us) server_negotiation.cc:407] Connection header received
0928 22:02:16.861563 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.861564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.861577 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.861627 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:16.862626 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.863132 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.864130 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.864348 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.864434 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.866730 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:16.866731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:16.866732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:16.866739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:16.868444 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.868929 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.868930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.868930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.868994 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.869272 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.869273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.869273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.869400 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:16.869461 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.869789 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.869852 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260928 22:02:16.870195 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.860585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:36421)
0928 22:02:16.860828 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:16.860865 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:16.861430 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:16.861701 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:16.861709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:16.862301 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:16.862521 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.862525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.863226 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.863227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.864035 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.864037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.864444 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.864445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.864576 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.865161 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:16.865170 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:16.866592 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:16.868534 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.868541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.868544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.868862 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.869063 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:16.869064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:16.869064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:16.869196 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:16.869460 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:16.869464 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:16.869640 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 22:02:16.869749 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:02:16.869917 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":145,"threads_started":1}
I20260928 22:02:16.870806 19089 heartbeater.cc:347] Connected to a master server at 127.2.165.126:35097
I20260928 22:02:16.870904 19089 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:16.871126 19089 heartbeater.cc:511] Master 127.2.165.126:35097 requested a full tablet report, sending...
I20260928 22:02:16.871527 18626 ts_manager.cc:194] Registered new tserver with Master: a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821)
I20260928 22:02:16.871893 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:36421
I20260928 22:02:16.874835 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.867583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.2.165.126:35097)
0928 22:02:16.867716 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.867720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.867861 (+ 141us) server_negotiation.cc:407] Connection header received
0928 22:02:16.867891 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.867892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.867905 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.867990 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:16.868590 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.869465 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.870542 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.870856 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.870943 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.872423 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:16.872424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:16.872424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:16.872432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:16.873883 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.874215 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.874216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.874216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.874268 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:16.874431 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:16.874431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:16.874432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:16.874541 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:16.874656 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.874709 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.874769 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":50,"threads_started":1}
I20260928 22:02:16.875628 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:02:16.878538 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.875976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.2.165.126:35097)
0928 22:02:16.876037 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.876039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.876174 (+ 135us) server_negotiation.cc:407] Connection header received
0928 22:02:16.876221 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.876221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.876248 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.876301 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:02:16.876400 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.876945 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.877745 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.877941 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.878058 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.878192 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:02:16.878282 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:02:16.878348 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.878414 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.878434 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:02:16.878885 18626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:02:16.880570 18626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:02:16.883020 18748 tablet_service.cc:1467] Processing CreateTablet for tablet f6072d581aa847c890f9cef42ab988ee (DEFAULT_TABLE table=test-table [id=8c9c938d1e924bb383ef482fdd645a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:16.883127 18887 tablet_service.cc:1467] Processing CreateTablet for tablet f6072d581aa847c890f9cef42ab988ee (DEFAULT_TABLE table=test-table [id=8c9c938d1e924bb383ef482fdd645a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:16.883255 18748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6072d581aa847c890f9cef42ab988ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:16.883312 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6072d581aa847c890f9cef42ab988ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:16.884828 18845 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Bootstrap starting.
I20260928 22:02:16.884950 18706 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap starting.
I20260928 22:02:16.885154 18845 tablet_bootstrap.cc:654] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:16.885273 18706 tablet_bootstrap.cc:654] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:16.885679 18845 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: No bootstrap required, opened a new log
I20260928 22:02:16.885730 18845 ts_tablet_manager.cc:1397] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:16.885941 18706 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: No bootstrap required, opened a new log
I20260928 22:02:16.885994 18706 ts_tablet_manager.cc:1397] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:16.886123 18845 raft_consensus.cc:348] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.886235 18845 raft_consensus.cc:374] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:16.886293 18845 raft_consensus.cc:729] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40a5ea73c3545ca82cccd746de55147, State: Initialized, Role: FOLLOWER
I20260928 22:02:16.886343 18845 consensus_queue.cc:260] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.886363 18706 raft_consensus.cc:348] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.886449 18845 ts_tablet_manager.cc:1428] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:16.886451 18706 raft_consensus.cc:374] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:16.886524 18706 raft_consensus.cc:729] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b65697fa17548c1b8cace90cbb099ad, State: Initialized, Role: FOLLOWER
I20260928 22:02:16.886582 18706 consensus_queue.cc:260] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.886693 18706 ts_tablet_manager.cc:1428] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:16.887749 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.882768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:40821 (local address 127.0.0.1:51354)
0928 22:02:16.882803 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:16.882816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:16.882855 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:16.883335 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:16.883336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:16.883339 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:16.883386 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.883388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.884697 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.884698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.886024 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:16.886027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.887522 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.887523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:16.887579 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.887599 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 22:02:16.887656 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:02:16.887694 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 22:02:16.887784 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:16.882891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.2.165.67:40821)
0928 22:02:16.883118 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:16.883124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:16.883135 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:02:16.883177 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:16.883179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:16.883196 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:16.883320 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:16.883458 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.884558 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.886683 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:16.887432 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:16.887493 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:16.887537 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:16.887687 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 22:02:16.887721 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":136,"threads_started":1}
I20260928 22:02:16.888453 19023 tablet_service.cc:1467] Processing CreateTablet for tablet f6072d581aa847c890f9cef42ab988ee (DEFAULT_TABLE table=test-table [id=8c9c938d1e924bb383ef482fdd645a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:16.888861 19023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6072d581aa847c890f9cef42ab988ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:16.891757 19098 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Bootstrap starting.
I20260928 22:02:16.892437 19098 tablet_bootstrap.cc:654] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:16.893011 19098 log.cc:824] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:16.893805 19098 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: No bootstrap required, opened a new log
I20260928 22:02:16.893908 19098 ts_tablet_manager.cc:1397] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:02:16.895414 19098 raft_consensus.cc:348] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.895619 19098 raft_consensus.cc:374] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:16.895668 19098 raft_consensus.cc:729] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f27afc473543dc9201dce918e1abf9, State: Initialized, Role: FOLLOWER
I20260928 22:02:16.895795 19098 consensus_queue.cc:260] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:16.896098 19098 ts_tablet_manager.cc:1428] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:02:16.896179 19089 heartbeater.cc:503] Master 127.2.165.126:35097 was elected leader, sending a full tablet report...
W20260928 22:02:16.970520 18954 tablet.cc:2369] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:02:17.077116 18815 tablet.cc:2369] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:02:17.110361 19090 tablet.cc:2369] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:02:17.221107 18957 raft_consensus.cc:482] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:17.221295 18957 raft_consensus.cc:504] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:17.221721 18957 leader_election.cc:288] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821), 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579)
I20260928 22:02:17.224843 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.222175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:42069 (local address 127.2.165.67:40821)
0928 22:02:17.222229 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:17.222231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:17.222241 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:02:17.222272 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:17.222273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:17.222283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:17.222371 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:17.222568 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.223060 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.224067 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.224435 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.224519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:17.224528 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:17.224729 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 22:02:17.224768 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:02:17.224843 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.221949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:40821 (local address 127.2.165.66:42069)
0928 22:02:17.221991 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:17.222004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:17.222068 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:17.222374 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:17.222375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:17.222377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:17.222460 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:17.222462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.223149 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.223150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:17.223972 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:17.223974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.224535 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.224536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:17.224614 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:17.224630 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:02:17.224728 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 22:02:17.224772 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 22:02:17.224977 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.222039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:37909 (local address 127.2.165.65:35579)
0928 22:02:17.222168 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:17.222170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:17.222281 (+ 111us) server_negotiation.cc:407] Connection header received
0928 22:02:17.222309 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:17.222310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:17.222333 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:17.222412 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:17.222602 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.223099 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.224088 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.224510 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.224600 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:17.224684 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:17.224854 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 22:02:17.224900 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 22:02:17.224999 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.221954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35579 (local address 127.2.165.66:37909)
0928 22:02:17.222180 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:17.222189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:17.222244 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:17.222433 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:17.222434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:17.222436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:17.222517 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:17.222518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.223207 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.223208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:17.223988 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:17.223990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.224641 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.224643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:17.224761 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:17.224765 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:17.224848 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:02:17.224888 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":124,"thread_start_us":53,"threads_started":1}
I20260928 22:02:17.225325 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6072d581aa847c890f9cef42ab988ee" candidate_uuid: "c40a5ea73c3545ca82cccd746de55147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b65697fa17548c1b8cace90cbb099ad" is_pre_election: true
I20260928 22:02:17.225492 18768 raft_consensus.cc:2454] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40a5ea73c3545ca82cccd746de55147 in term 0.
I20260928 22:02:17.225481 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6072d581aa847c890f9cef42ab988ee" candidate_uuid: "c40a5ea73c3545ca82cccd746de55147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f27afc473543dc9201dce918e1abf9" is_pre_election: true
I20260928 22:02:17.225647 19043 raft_consensus.cc:2454] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40a5ea73c3545ca82cccd746de55147 in term 0.
I20260928 22:02:17.225715 18839 leader_election.cc:302] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad, c40a5ea73c3545ca82cccd746de55147; no voters:
I20260928 22:02:17.225875 18957 raft_consensus.cc:2781] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:02:17.225939 18957 raft_consensus.cc:482] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:02:17.225965 18957 raft_consensus.cc:3037] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:17.226727 18957 raft_consensus.cc:504] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:17.226872 18957 leader_election.cc:288] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [CANDIDATE]: Term 1 election: Requested vote from peers a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821), 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579)
I20260928 22:02:17.227016 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6072d581aa847c890f9cef42ab988ee" candidate_uuid: "c40a5ea73c3545ca82cccd746de55147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b65697fa17548c1b8cace90cbb099ad"
I20260928 22:02:17.227054 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6072d581aa847c890f9cef42ab988ee" candidate_uuid: "c40a5ea73c3545ca82cccd746de55147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f27afc473543dc9201dce918e1abf9"
I20260928 22:02:17.227130 18768 raft_consensus.cc:3037] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:17.227126 19043 raft_consensus.cc:3037] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:17.227834 18768 raft_consensus.cc:2454] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40a5ea73c3545ca82cccd746de55147 in term 1.
I20260928 22:02:17.227895 19043 raft_consensus.cc:2454] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40a5ea73c3545ca82cccd746de55147 in term 1.
I20260928 22:02:17.228055 18839 leader_election.cc:302] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad, c40a5ea73c3545ca82cccd746de55147; no voters:
I20260928 22:02:17.228169 18957 raft_consensus.cc:2781] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:17.228333 18957 raft_consensus.cc:686] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 1 LEADER]: Becoming Leader. State: Replica: c40a5ea73c3545ca82cccd746de55147, State: Running, Role: LEADER
I20260928 22:02:17.228410 18957 consensus_queue.cc:237] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:17.229328 18626 catalog_manager.cc:5495] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 reported cstate change: term changed from 0 to 1, leader changed from <none> to c40a5ea73c3545ca82cccd746de55147 (127.2.165.66). New cstate: current_term: 1 leader_uuid: "c40a5ea73c3545ca82cccd746de55147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } health_report { overall_health: HEALTHY } } }
I20260928 22:02:17.242692 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.240368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.2.165.66:43427)
0928 22:02:17.240437 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:17.240439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:17.240449 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:02:17.240495 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:17.240495 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:17.240529 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:17.240593 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:02:17.240758 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.241251 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.242057 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.242269 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.242325 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:17.242462 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:02:17.242517 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 22:02:17.242636 (+ 119us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41500: token expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 22:02:17.242816 18959 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41500: token expired
I20260928 22:02:17.249150 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.243173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.2.165.126:35097)
0928 22:02:17.243224 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:17.243226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:17.243250 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:02:17.243309 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:17.243310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:17.243318 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:17.243353 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:17.243825 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.244298 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.245018 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.245214 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.245272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:17.246514 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:17.246516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:17.246516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:17.246523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:17.247945 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:17.248327 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:17.248328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:17.248329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:17.248385 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:17.248557 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:17.248558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:17.248558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:17.248726 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:17.248817 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:17.248924 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:02:17.248997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:02:17.252323 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.250092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.2.165.66:43427)
0928 22:02:17.250155 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:17.250157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:17.250175 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:02:17.250201 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:17.250202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:17.250211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:17.250266 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:02:17.250366 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.250831 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.251532 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:17.251752 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:17.251823 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:17.251971 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:02:17.252081 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:02:17.252137 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:17.252206 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:02:17.252227 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:02:17.253970 18768 raft_consensus.cc:1261] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Refusing update from remote peer c40a5ea73c3545ca82cccd746de55147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:17.253971 19043 raft_consensus.cc:1261] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 1 FOLLOWER]: Refusing update from remote peer c40a5ea73c3545ca82cccd746de55147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:17.254320 18957 consensus_queue.cc:1035] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:17.254457 19103 consensus_queue.cc:1035] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:17.256523 19106 mvcc.cc:205] Tried to move back new op lower bound from 7334432510989537280 to 7334432510888382464. Current Snapshot: MvccSnapshot[applied={T|T < 7334432510989537280}]
I20260928 22:02:17.259341 18626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:17.259454 18626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:17.260253 18626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Sending DeleteTablet for 3 replicas of tablet f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:17.260592 18748 tablet_service.cc:1514] Processing DeleteTablet for tablet f6072d581aa847c890f9cef42ab988ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:17 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:57434
I20260928 22:02:17.260630 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18664
I20260928 22:02:17.261164 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet f6072d581aa847c890f9cef42ab988ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:17 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:41496
I20260928 22:02:17.261369 19116 tablet_replica.cc:330] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: stopping tablet replica
I20260928 22:02:17.261431 19116 raft_consensus.cc:2229] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:02:17.261552 19116 raft_consensus.cc:2258] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:17.261651 19023 tablet_service.cc:1514] Processing DeleteTablet for tablet f6072d581aa847c890f9cef42ab988ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:17 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:51354
I20260928 22:02:17.261945 19117 tablet_replica.cc:330] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: stopping tablet replica
I20260928 22:02:17.261953 19116 ts_tablet_manager.cc:1905] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:17.262109 19117 raft_consensus.cc:2229] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:17.262264 19117 raft_consensus.cc:2258] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:17.262756 19117 ts_tablet_manager.cc:1905] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:17.264012 19116 ts_tablet_manager.cc:1918] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:17.264138 19116 log.cc:1196] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:17.264441 19116 ts_tablet_manager.cc:1939] T f6072d581aa847c890f9cef42ab988ee P c40a5ea73c3545ca82cccd746de55147: Deleting consensus metadata
I20260928 22:02:17.264853 19117 ts_tablet_manager.cc:1918] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:17.264864 18613 catalog_manager.cc:4841] TS c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427): tablet f6072d581aa847c890f9cef42ab988ee (table test-table [id=8c9c938d1e924bb383ef482fdd645a2d]) successfully deleted
I20260928 22:02:17.264997 19117 log.cc:1196] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:17.265328 19117 ts_tablet_manager.cc:1939] T f6072d581aa847c890f9cef42ab988ee P a4f27afc473543dc9201dce918e1abf9: Deleting consensus metadata
I20260928 22:02:17.265754 18611 catalog_manager.cc:4841] TS a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821): tablet f6072d581aa847c890f9cef42ab988ee (table test-table [id=8c9c938d1e924bb383ef482fdd645a2d]) successfully deleted
W20260928 22:02:17.267730 18610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:35579 (error 108)
I20260928 22:02:17.267973 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18820
I20260928 22:02:17.268639 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:17.268113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:35579 (local address 127.0.0.1:57440)
0928 22:02:17.268553 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:17.268599 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:35579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419}
I20260928 22:02:17.272643 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18958
W20260928 22:02:17.272558 18610 catalog_manager.cc:4584] TS 7b65697fa17548c1b8cace90cbb099ad (127.2.165.65:35579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6072d581aa847c890f9cef42ab988ee: Network error: Client connection negotiation failed: client connection to 127.2.165.65:35579: connect: Connection refused (error 111)
I20260928 22:02:17.277542 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 18593
I20260928 22:02:37.283174 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35097
--webserver_interface=127.2.165.126
--webserver_port=38289
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:35097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:37.365705 19119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:37.366106 19119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:37.366163 19119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:37.367551 19119 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:02:37.367614 19119 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:02:37.367635 19119 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:02:37.367654 19119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:02:37.367686 19119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:37.367724 19119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:37.367756 19119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:02:37.367787 19119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:02:37.369439 19119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:35097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:37.369747 19119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:37.370002 19119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:37.375609 19125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:37.375739 19119 server_base.cc:1034] running on GCE node
W20260928 22:02:37.375614 19127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:37.375700 19124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:37.376099 19119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:37.376413 19119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:37.377586 19119 hybrid_clock.cc:648] HybridClock initialized: now 1790632957377549 us; error 47 us; skew 500 ppm
Sep 28 22:02:37 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:37.380177 19119 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:02:37.380806 19119 webserver.cc:460] Webserver started at http://127.2.165.126:38289/ using document root <none> and password file <none>
I20260928 22:02:37.381052 19119 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:37.381099 19119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:37.382612 19119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:02:37.383431 19134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.383652 19119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:37.383751 19119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "a6ff3bff1ec344a3a768503366c0ef4f"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "5ebef2410eba03b34b2d55aebff2f725"
server_key_iv: "13994c6a1b77d7c13fe935ee62eb9d83"
server_key_version: "encryptionkey@0"
I20260928 22:02:37.384109 19119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:37.399495 19119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:37.402004 19119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:37.402124 19119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:37.406133 19119 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:35097
I20260928 22:02:37.406208 19186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:35097 every 8 connection(s)
I20260928 22:02:37.406571 19119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:02:37.408104 19187 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:02:37.408649 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 19119
I20260928 22:02:37.408947 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Bootstrap starting.
I20260928 22:02:37.409029 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:35579
--local_ip_for_outbound_sockets=127.2.165.65
--tserver_master_addrs=127.2.165.126:35097
--webserver_port=40191
--webserver_interface=127.2.165.65
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:02:37.418181 19187 log.cc:824] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:37.420526 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:02:37.420917 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: Bootstrap complete.
I20260928 22:02:37.423179 19187 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:37.423439 19187 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ff3bff1ec344a3a768503366c0ef4f, State: Initialized, Role: FOLLOWER
I20260928 22:02:37.423625 19187 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:37.423704 19187 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:02:37.423786 19187 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:02:37.423838 19187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:02:37.424888 19187 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:37.425010 19187 leader_election.cc:302] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ff3bff1ec344a3a768503366c0ef4f; no voters:
I20260928 22:02:37.425210 19187 leader_election.cc:288] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:02:37.425256 19191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:02:37.425438 19191 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [term 3 LEADER]: Becoming Leader. State: Replica: a6ff3bff1ec344a3a768503366c0ef4f, State: Running, Role: LEADER
I20260928 22:02:37.425498 19187 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:02:37.425611 19191 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } }
I20260928 22:02:37.425882 19191 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ff3bff1ec344a3a768503366c0ef4f. Latest consensus state: current_term: 3 leader_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } } }
I20260928 22:02:37.425886 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ff3bff1ec344a3a768503366c0ef4f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 35097 } } }
I20260928 22:02:37.425962 19191 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:37.426129 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:37.427416 19207 catalog_manager.cc:1259] Loaded cluster ID: eab90900cd114589a304b776f787c1f4
I20260928 22:02:37.427486 19207 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: loading cluster ID for follower catalog manager: success
I20260928 22:02:37.428329 19207 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6ff3bff1ec344a3a768503366c0ef4f: acquiring CA information for follower catalog manager: success
F20260928 22:02:37.428761 19207 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f07fc6cd9ad google::LogMessage::Fail() at ??:0
@ 0x7f07fc6d088a google::LogMessage::SendToLog() at ??:0
@ 0x7f07fc6cd4de google::LogMessage::Flush() at ??:0
@ 0x7f07fc6cea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f07fee3a104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f08046266b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f0804626989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f0804626e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f08046298c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f07fe590954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f07ff3f46db start_thread at ??:0
@ 0x7f07fb1cb61f clone at ??:0
W20260928 22:02:37.527828 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:37.528149 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:37.528190 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:37.530088 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:37.530165 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:37.530211 19189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:02:37.532543 19189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:35579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=40191
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:37.532871 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:37.533181 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:37.538942 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:37.538915 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:37.539127 19189 server_base.cc:1034] running on GCE node
W20260928 22:02:37.538923 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:37.539438 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:37.539716 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:37.540896 19189 hybrid_clock.cc:648] HybridClock initialized: now 1790632957540880 us; error 35 us; skew 500 ppm
Sep 28 22:02:37 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:37.544256 19189 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:02:37.544896 19189 webserver.cc:460] Webserver started at http://127.2.165.65:40191/ using document root <none> and password file <none>
I20260928 22:02:37.545194 19189 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:37.545269 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:37.547108 19189 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:02:37.547988 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.548214 19189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:37.548354 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "7b65697fa17548c1b8cace90cbb099ad"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "93f4e520cdb0e778526d7e374212b945"
server_key_iv: "c462f98df167c01ba6b56b9cd37b46d9"
server_key_version: "encryptionkey@0"
I20260928 22:02:37.548774 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:37.557711 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:37.560413 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:37.560566 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:37.560853 19189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:37.561446 19242 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:02:37.563521 19189 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:02:37.563609 19189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:02:37.563658 19189 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:02:37.564375 19189 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:02:37.564437 19189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:37.564505 19242 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap starting.
I20260928 22:02:37.572108 19189 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:35579
I20260928 22:02:37.572561 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:02:37.575318 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 19189
I20260928 22:02:37.575819 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:43427
--local_ip_for_outbound_sockets=127.2.165.66
--tserver_master_addrs=127.2.165.126:35097
--webserver_port=41321
--webserver_interface=127.2.165.66
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:02:37.576025 19242 log.cc:824] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:37.576431 19349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:35579 every 8 connection(s)
I20260928 22:02:37.579035 19242 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:02:37.579389 19242 tablet_bootstrap.cc:492] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Bootstrap complete.
I20260928 22:02:37.580642 19242 ts_tablet_manager.cc:1397] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260928 22:02:37.582226 19242 raft_consensus.cc:348] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:37.582623 19242 raft_consensus.cc:729] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b65697fa17548c1b8cace90cbb099ad, State: Initialized, Role: FOLLOWER
I20260928 22:02:37.582830 19242 consensus_queue.cc:260] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:37.583165 19242 ts_tablet_manager.cc:1428] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 22:02:37.655567 19352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:02:37.573379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:48789)
0928 22:02:37.573562 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.573610 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:37.574130 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:37.655342 (+ 81212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
W20260928 22:02:37.655864 19350 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:35097 (0 consecutive failures): Network error: Failed to ping master at 127.2.165.126:35097: Client connection negotiation failed: client connection to 127.2.165.126:35097: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:02:37.656255 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.656112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:41529)
0928 22:02:37.656172 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.656225 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:02:37.656750 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.656610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:44253)
0928 22:02:37.656667 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.656720 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 22:02:37.656890 19350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:02:37.669570 19354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:37.669795 19354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:37.669822 19354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:37.671125 19354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:37.671172 19354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:37.671203 19354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:02:37.672964 19354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:43427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=41321
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:37.673170 19354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:37.673374 19354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:37.679168 19360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:37.679204 19354 server_base.cc:1034] running on GCE node
W20260928 22:02:37.679168 19363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:37.679174 19361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:37.679646 19354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:37.679952 19354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:37.681159 19354 hybrid_clock.cc:648] HybridClock initialized: now 1790632957681120 us; error 29 us; skew 500 ppm
Sep 28 22:02:37 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:37.683954 19354 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:02:37.684501 19354 webserver.cc:460] Webserver started at http://127.2.165.66:41321/ using document root <none> and password file <none>
I20260928 22:02:37.684825 19354 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:37.684895 19354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:37.686347 19354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:37.687383 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.687625 19354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:37.687745 19354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "c40a5ea73c3545ca82cccd746de55147"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "c3bb5ca946a97d0292011034d1eb5858"
server_key_iv: "f8023e2e6cc8eaa1e25e733e04836e9b"
server_key_version: "encryptionkey@0"
I20260928 22:02:37.688158 19354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:37.709910 19354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:37.712405 19354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:37.712592 19354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:37.712894 19354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:37.713245 19354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:37.713302 19354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.713348 19354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:37.713407 19354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.719755 19354 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:43427
I20260928 22:02:37.719806 19483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:43427 every 8 connection(s)
I20260928 22:02:37.720178 19354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:02:37.721343 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.720937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:47175)
0928 22:02:37.721103 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.721181 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260928 22:02:37.721819 19484 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:35097 (0 consecutive failures): Network error: Failed to ping master at 127.2.165.126:35097: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
I20260928 22:02:37.722195 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.722049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:38269)
0928 22:02:37.722110 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.722148 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:02:37.722210 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 19354
I20260928 22:02:37.722566 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:40821
--local_ip_for_outbound_sockets=127.2.165.67
--tserver_master_addrs=127.2.165.126:35097
--webserver_port=33609
--webserver_interface=127.2.165.67
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:02:37.722599 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.722485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:33953)
0928 22:02:37.722541 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.722574 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 22:02:37.722818 19484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:02:37.807955 19487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:37.808348 19487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:37.808422 19487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:37.809648 19487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:37.809721 19487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:37.809768 19487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:02:37.811362 19487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:40821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=33609
--tserver_master_addrs=127.2.165.126:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:37.811646 19487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:37.811915 19487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:37.817117 19492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:37.817124 19495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:37.817139 19493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:37.817360 19487 server_base.cc:1034] running on GCE node
I20260928 22:02:37.817580 19487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:37.817870 19487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:37.819041 19487 hybrid_clock.cc:648] HybridClock initialized: now 1790632957819027 us; error 37 us; skew 500 ppm
Sep 28 22:02:37 dist-test-slave-3t36 krb5kdc[18045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:37.822057 19487 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:02:37.822667 19487 webserver.cc:460] Webserver started at http://127.2.165.67:33609/ using document root <none> and password file <none>
I20260928 22:02:37.822926 19487 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:37.822978 19487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:37.824489 19487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:02:37.825232 19502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.825448 19487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:37.825562 19487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "a4f27afc473543dc9201dce918e1abf9"
format_stamp: "Formatted at 2026-09-28 22:01:55 on dist-test-slave-3t36"
server_key: "8085888e2149e670e306d05eecccdcb7"
server_key_iv: "6e79b735911373e61073d021031323e4"
server_key_version: "encryptionkey@0"
I20260928 22:02:37.825937 19487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:37.853024 19487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:37.855468 19487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:37.855664 19487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:37.855943 19487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:37.856273 19487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:37.856326 19487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.856382 19487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:37.856407 19487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:37.863062 19487 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:40821
I20260928 22:02:37.863106 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:40821 every 8 connection(s)
I20260928 22:02:37.863490 19487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:02:37.864573 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.864211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:52321)
0928 22:02:37.864361 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.864433 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
W20260928 22:02:37.865037 19616 heartbeater.cc:644] Failed to heartbeat to 127.2.165.126:35097 (0 consecutive failures): Network error: Failed to ping master at 127.2.165.126:35097: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
I20260928 22:02:37.865337 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.865218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:46571)
0928 22:02:37.865293 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.865319 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 22:02:37.865742 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:37.865638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:45721)
0928 22:02:37.865685 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:37.865719 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:02:37.865868 19616 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:02:37.868129 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 19487
I20260928 22:02:38.657537 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:38.657268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:41285)
0928 22:02:38.657427 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:38.657474 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260928 22:02:38.723467 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:38.723195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:47721)
0928 22:02:38.723371 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:38.723412 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
I20260928 22:02:38.866614 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:38.866341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:41597)
0928 22:02:38.866515 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:38.866557 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260928 22:02:39.178965 19623 raft_consensus.cc:482] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:39.179098 19623 raft_consensus.cc:504] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:39.179519 19623 leader_election.cc:288] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821), c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
Sep 28 22:02:39 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.66@KRBTEST.COM
Sep 28 22:02:39 dist-test-slave-3t36 krb5kdc[18045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.67@KRBTEST.COM
I20260928 22:02:39.190687 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:39.179700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:40821 (local address 127.2.165.65:51653)
0928 22:02:39.179822 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:39.179835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:39.179894 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:39.180794 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:39.180795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:39.181195 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:39.181373 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:39.181381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.182653 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.182654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:39.183405 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:39.183407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.184038 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.184039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:39.184110 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:39.184842 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:39.184847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:39.186630 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:39.188962 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:39.188969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:39.188971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:39.189203 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:39.189480 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:39.189481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:39.189482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:39.189599 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:39.190148 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:39.190154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:39.190261 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 22:02:39.190346 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:02:39.190537 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":1,"thread_start_us":48,"threads_started":1}
I20260928 22:02:39.190770 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:39.179836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:51653 (local address 127.2.165.67:40821)
0928 22:02:39.180008 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:39.180033 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:39.180069 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:02:39.180528 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:39.180530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:39.180568 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:39.180811 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:39.181526 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.182543 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.183508 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.183933 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.184072 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:39.186736 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:39.186742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:39.186742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:39.186756 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:39.188869 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:39.189292 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:39.189296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:39.189296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:39.189390 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:39.189674 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:39.189675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:39.189676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:39.190080 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:39.190168 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:39.190463 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 22:02:39.190632 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260928 22:02:39.190909 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:39.179835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:39855 (local address 127.2.165.66:43427)
0928 22:02:39.180036 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:39.180045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:39.180075 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:02:39.180516 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:39.180518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:39.180559 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:39.180742 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:39.181503 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.182464 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.183480 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.183882 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.184010 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:39.186440 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:39.186446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:39.186446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:39.186459 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:39.188863 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:39.189710 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:39.189713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:39.189713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:39.189816 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:39.190022 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:39.190023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:39.190023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:39.190311 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:39.190411 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:39.190601 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 22:02:39.190721 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260928 22:02:39.190759 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:39.179700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:43427 (local address 127.2.165.65:39855)
0928 22:02:39.179823 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:39.179835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:39.179911 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:39.180781 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:39.180784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:39.181192 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:39.181374 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:39.181381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.182566 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.182567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:39.183388 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:39.183391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:39.183985 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:39.183986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:39.184108 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:39.184634 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:39.184643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:39.186211 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:39.189273 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:39.189273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:39.189274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:39.189401 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:39.189898 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:39.189898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:39.189899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:39.189953 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:39.190409 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:39.190410 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:39.190454 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 22:02:39.190530 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:02:39.190606 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":53,"thread_start_us":48,"threads_started":1}
W20260928 22:02:39.191852 19237 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
W20260928 22:02:39.192034 19239 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:39.192116 19239 leader_election.cc:302] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad; no voters: a4f27afc473543dc9201dce918e1abf9, c40a5ea73c3545ca82cccd746de55147
I20260928 22:02:39.192229 19623 raft_consensus.cc:2726] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:02:39.658208 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:39.658037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:37289)
0928 22:02:39.658123 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:39.658165 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 22:02:39.724205 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:39.724005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:56375)
0928 22:02:39.724126 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:39.724164 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260928 22:02:39.867412 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:39.867186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:38705)
0928 22:02:39.867322 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:39.867363 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260928 22:02:40.659040 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:40.658778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:44039)
0928 22:02:40.658945 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:40.658992 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260928 22:02:40.724977 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:40.724761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:60181)
0928 22:02:40.724889 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:40.724931 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260928 22:02:40.868171 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:40.867936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:38353)
0928 22:02:40.868091 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:40.868127 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260928 22:02:40.892130 19633 raft_consensus.cc:482] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:40.892243 19633 raft_consensus.cc:504] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:40.892448 19633 leader_election.cc:288] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821), c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
W20260928 22:02:40.892927 19237 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
W20260928 22:02:40.893014 19239 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:40.893110 19239 leader_election.cc:302] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad; no voters: a4f27afc473543dc9201dce918e1abf9, c40a5ea73c3545ca82cccd746de55147
I20260928 22:02:40.893224 19633 raft_consensus.cc:2726] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:02:41.660061 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:41.659779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:38051)
0928 22:02:41.659967 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:41.660013 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
I20260928 22:02:41.725780 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:41.725595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:50225)
0928 22:02:41.725705 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:41.725734 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260928 22:02:41.869024 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:41.868781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:34175)
0928 22:02:41.868926 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:41.868972 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260928 22:02:42.660900 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:42.660621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:53481)
0928 22:02:42.660806 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:42.660852 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260928 22:02:42.726652 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:42.726285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:55927)
0928 22:02:42.726529 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:42.726582 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260928 22:02:42.869771 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:42.869523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:56703)
0928 22:02:42.869671 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:42.869722 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260928 22:02:43.317516 19640 raft_consensus.cc:482] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:43.317644 19640 raft_consensus.cc:504] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:43.317826 19640 leader_election.cc:288] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821), c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
W20260928 22:02:43.318264 19237 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
W20260928 22:02:43.318377 19239 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:43.318473 19239 leader_election.cc:302] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad; no voters: a4f27afc473543dc9201dce918e1abf9, c40a5ea73c3545ca82cccd746de55147
I20260928 22:02:43.318557 19640 raft_consensus.cc:2726] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:02:43.661655 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:43.661428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:34307)
0928 22:02:43.661572 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:43.661607 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260928 22:02:43.727387 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:43.727136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:42925)
0928 22:02:43.727285 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:43.727330 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260928 22:02:43.870572 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:43.870322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:50851)
0928 22:02:43.870467 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:43.870515 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 22:02:44.662400 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:44.662135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:43663)
0928 22:02:44.662290 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:44.662350 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260928 22:02:44.728163 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:44.727904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:32883)
0928 22:02:44.728077 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:44.728115 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260928 22:02:44.871444 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:44.871135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:59099)
0928 22:02:44.871300 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:44.871398 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260928 22:02:45.663168 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:45.662936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:47419)
0928 22:02:45.663074 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:45.663117 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260928 22:02:45.729036 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:45.728805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:37925)
0928 22:02:45.728947 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:45.728990 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260928 22:02:45.872246 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:45.872007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:38663)
0928 22:02:45.872154 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:45.872196 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 22:02:46.460407 19650 raft_consensus.cc:482] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:46.460546 19650 raft_consensus.cc:504] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:46.460786 19650 leader_election.cc:288] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821), c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
W20260928 22:02:46.461165 19237 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
W20260928 22:02:46.461254 19239 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:46.461323 19239 leader_election.cc:302] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad; no voters: a4f27afc473543dc9201dce918e1abf9, c40a5ea73c3545ca82cccd746de55147
I20260928 22:02:46.461414 19650 raft_consensus.cc:2726] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:02:46.663930 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:46.663676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:49795)
0928 22:02:46.663834 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:46.663881 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260928 22:02:46.729796 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:46.729554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:55045)
0928 22:02:46.729701 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:46.729748 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260928 22:02:46.873139 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:46.872885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:50913)
0928 22:02:46.873046 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:46.873090 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260928 22:02:47.664889 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:47.664553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:60725)
0928 22:02:47.664794 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:47.664841 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260928 22:02:47.730494 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:47.730247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:38241)
0928 22:02:47.730410 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:47.730448 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260928 22:02:47.873967 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:47.873659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:36209)
0928 22:02:47.873823 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:47.873867 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260928 22:02:48.665724 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:48.665480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:35175)
0928 22:02:48.665625 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:48.665675 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 22:02:48.731313 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:48.731069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:53133)
0928 22:02:48.731217 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:48.731267 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260928 22:02:48.874882 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:48.874588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:38209)
0928 22:02:48.874747 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:48.874811 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260928 22:02:49.666483 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:49.666230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:56343)
0928 22:02:49.666384 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:49.666427 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260928 22:02:49.732134 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:49.731887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:40619)
0928 22:02:49.732046 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:49.732087 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260928 22:02:49.875725 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:49.875467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:39383)
0928 22:02:49.875630 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:49.875674 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260928 22:02:50.642490 19663 raft_consensus.cc:482] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:50.642617 19663 raft_consensus.cc:504] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f27afc473543dc9201dce918e1abf9" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40821 } } peers { permanent_uuid: "7b65697fa17548c1b8cace90cbb099ad" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 35579 } } peers { permanent_uuid: "c40a5ea73c3545ca82cccd746de55147" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 43427 } }
I20260928 22:02:50.642786 19663 leader_election.cc:288] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821), c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427)
W20260928 22:02:50.643268 19237 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4f27afc473543dc9201dce918e1abf9 (127.2.165.67:40821): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
W20260928 22:02:50.643359 19239 leader_election.cc:341] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c40a5ea73c3545ca82cccd746de55147 (127.2.165.66:43427): Not found: Tablet not found: f6072d581aa847c890f9cef42ab988ee
I20260928 22:02:50.643399 19239 leader_election.cc:302] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b65697fa17548c1b8cace90cbb099ad; no voters: a4f27afc473543dc9201dce918e1abf9, c40a5ea73c3545ca82cccd746de55147
I20260928 22:02:50.643508 19663 raft_consensus.cc:2726] T f6072d581aa847c890f9cef42ab988ee P 7b65697fa17548c1b8cace90cbb099ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:02:50.667331 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:50.667043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:58045)
0928 22:02:50.667246 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:50.667284 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":139,"threads_started":1}
I20260928 22:02:50.733044 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:50.732793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:38405)
0928 22:02:50.732941 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:50.732987 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
I20260928 22:02:50.876466 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:50.876217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:51915)
0928 22:02:50.876365 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:50.876409 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260928 22:02:51.668203 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:51.667919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:36265)
0928 22:02:51.668102 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:51.668153 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260928 22:02:51.733834 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:51.733612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:59213)
0928 22:02:51.733741 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:51.733785 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260928 22:02:51.877243 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:51.877033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:40189)
0928 22:02:51.877157 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:51.877194 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260928 22:02:52.669200 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:52.668944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.65:44723)
0928 22:02:52.669095 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:52.669151 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 22:02:52.734721 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:52.734505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.66:46487)
0928 22:02:52.734637 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:52.734677 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260928 22:02:52.876396 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 19189
I20260928 22:02:52.878046 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:52.877806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:35097 (local address 127.2.165.67:40001)
0928 22:02:52.877944 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:52.877989 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.126:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260928 22:02:52.881414 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 19354
I20260928 22:02:52.886037 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 19487
2026-09-28T22:02:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58724 ms)
[----------] 30 tests from SecurityITest (242632 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:02:52 dist-test-slave-3t36 krb5kdc[19675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:02:52 dist-test-slave-3t36 krb5kdc[19675](info): set up 2 sockets
Sep 28 22:02:52 dist-test-slave-3t36 krb5kdc[19675](info): commencing operation
krb5kdc: starting...
W20260928 22:02:53.940770 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:02:54.076303 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:44241
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:44241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:54.161288 19688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:54.161587 19688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:54.161635 19688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:54.162911 19688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:02:54.162997 19688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:54.163024 19688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:54.163057 19688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:02:54.163080 19688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:02:54.164637 19688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:44241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:54.164969 19688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:54.165215 19688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:02:54.171115 19688 server_base.cc:1034] running on GCE node
W20260928 22:02:54.171022 19694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:54.171065 19696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:54.171036 19693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:54.171631 19688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:54.171959 19688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:54.173117 19688 hybrid_clock.cc:648] HybridClock initialized: now 1790632974173090 us; error 45 us; skew 500 ppm
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:54.175719 19688 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:02:54.176229 19688 webserver.cc:460] Webserver started at http://127.2.165.126:33499/ using document root <none> and password file <none>
I20260928 22:02:54.176422 19688 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:54.176465 19688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:54.176560 19688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:54.177690 19688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "2fa895006fb2bd75ad6dce2b9ec184f4"
server_key_iv: "0be3ae8c0b64e3ac222be8097ab33500"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.178009 19688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "2fa895006fb2bd75ad6dce2b9ec184f4"
server_key_iv: "0be3ae8c0b64e3ac222be8097ab33500"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.179477 19688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:02:54.180433 19703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.180749 19688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:54.180874 19688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "2fa895006fb2bd75ad6dce2b9ec184f4"
server_key_iv: "0be3ae8c0b64e3ac222be8097ab33500"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.180967 19688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:54.192408 19688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:54.195042 19688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:54.195242 19688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:54.199695 19688 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:44241
I20260928 22:02:54.199753 19755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:44241 every 8 connection(s)
I20260928 22:02:54.200119 19688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:02:54.201292 19756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:54.201895 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 19688
I20260928 22:02:54.202049 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:02:54.202248 2709 external_mini_cluster.cc:1428] Setting key 0582bf2a4598975f8747e401b4ebaede
I20260928 22:02:54.204811 19756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9: Bootstrap starting.
I20260928 22:02:54.205488 19756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:54.205789 19756 log.cc:824] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:54.206670 19756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9: No bootstrap required, opened a new log
I20260928 22:02:54.208976 19756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44241 } }
I20260928 22:02:54.209131 19756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:54.209153 19756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b19ecc65c1d4cba9bfd498e5f3943c9, State: Initialized, Role: FOLLOWER
I20260928 22:02:54.209297 19756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44241 } }
I20260928 22:02:54.209396 19756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:02:54.209429 19756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:02:54.209481 19756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:54.210290 19756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44241 } }
I20260928 22:02:54.210449 19756 leader_election.cc:302] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b19ecc65c1d4cba9bfd498e5f3943c9; no voters:
I20260928 22:02:54.210667 19756 leader_election.cc:288] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:02:54.210757 19761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:54.210966 19756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:02:54.210984 19761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2b19ecc65c1d4cba9bfd498e5f3943c9, State: Running, Role: LEADER
I20260928 22:02:54.211122 19761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44241 } }
I20260928 22:02:54.211486 19762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44241 } } }
I20260928 22:02:54.211609 19762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:54.211490 19761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b19ecc65c1d4cba9bfd498e5f3943c9. Latest consensus state: current_term: 1 leader_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19ecc65c1d4cba9bfd498e5f3943c9" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44241 } } }
I20260928 22:02:54.211994 19761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9 [sys.catalog]: This master's current role is: LEADER
W20260928 22:02:54.213066 19776 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:02:54.213172 19776 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:02:54.213254 19770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:02:54.213446 19770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:02:54.215065 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.202860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.2.165.126:44241)
0928 22:02:54.203073 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.203081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.203109 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:02:54.203557 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.203565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.203615 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.203741 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:54.204651 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.205753 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.206866 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.207489 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.207614 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.210437 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:54.210442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:54.210443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:54.210457 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:54.212956 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.213484 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.213488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.213488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.213611 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.213976 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.213977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.213978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.214343 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:54.214519 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.214658 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.214835 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260928 22:02:54.215277 19770 catalog_manager.cc:1347] Generated new cluster ID: b8844af8a2d74f509152050e1a88df88
I20260928 22:02:54.215327 19770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:02:54.236020 19770 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:02:54.236934 19770 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:02:54.240077 19770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9: Generated new TSK 0
I20260928 22:02:54.240304 19770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 22:02:54.280108 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44241
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:54.398268 19782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:54.398880 19782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:54.398983 19782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:54.401191 19782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:54.401261 19782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:54.401310 19782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:02:54.403124 19782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:54.403460 19782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:54.403795 19782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:54.410617 19788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:54.410609 19790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:54.410761 19787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:54.411043 19782 server_base.cc:1034] running on GCE node
I20260928 22:02:54.411257 19782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:54.411582 19782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:54.412755 19782 hybrid_clock.cc:648] HybridClock initialized: now 1790632974412739 us; error 28 us; skew 500 ppm
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:54.415632 19782 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:02:54.416155 19782 webserver.cc:460] Webserver started at http://127.2.165.65:39623/ using document root <none> and password file <none>
I20260928 22:02:54.416419 19782 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:54.416498 19782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:54.416653 19782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:54.417822 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "edb5ee2c92794da294c826024a01448b"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "6bb2fafc12333c0eb656e9a7e7680bb6"
server_key_iv: "a7ca7caeca92abd72f4dc4374ae17fe1"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.418197 19782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "edb5ee2c92794da294c826024a01448b"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "6bb2fafc12333c0eb656e9a7e7680bb6"
server_key_iv: "a7ca7caeca92abd72f4dc4374ae17fe1"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.419932 19782 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 22:02:54.420945 19797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.421144 19782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:54.421265 19782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "edb5ee2c92794da294c826024a01448b"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "6bb2fafc12333c0eb656e9a7e7680bb6"
server_key_iv: "a7ca7caeca92abd72f4dc4374ae17fe1"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.421350 19782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:54.469384 19782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:54.472028 19782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:54.472213 19782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:54.472474 19782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:54.472843 19782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:54.472904 19782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.472947 19782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:54.472976 19782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.479471 19782 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:39399
I20260928 22:02:54.479553 19910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:39399 every 8 connection(s)
I20260928 22:02:54.479900 19782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:54.488359 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 19782
I20260928 22:02:54.488458 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:02:54.488761 2709 external_mini_cluster.cc:1428] Setting key 4198d0d6381916249c7cc38dcd42219c
I20260928 22:02:54.490573 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.480725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:46819 (local address 127.2.165.126:44241)
0928 22:02:54.480788 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.480790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.481007 (+ 217us) server_negotiation.cc:407] Connection header received
0928 22:02:54.481556 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.481557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.481570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.481622 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:54.482642 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.483207 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.484310 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.484589 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.484704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.487086 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:54.487088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:54.487088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:54.487096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:54.488871 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.489405 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.489406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.489406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.489475 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.489839 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.489839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.489840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.489991 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:54.490049 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.490422 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.490507 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:02:54.490906 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.480646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:44241 (local address 127.2.165.65:46819)
0928 22:02:54.480873 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:54.480925 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:54.481439 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:54.481696 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:54.481700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:54.482304 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:54.482530 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.482535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.483322 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.483323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.484218 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.484220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.484726 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.484728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.484868 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.485479 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:54.485488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:54.486975 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:54.488989 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:54.488997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:54.488999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:54.489302 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:54.489594 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:54.489596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:54.489596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:54.489760 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:54.490065 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:54.490072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:54.490268 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 22:02:54.490388 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 22:02:54.490597 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260928 22:02:54.491474 19911 heartbeater.cc:347] Connected to a master server at 127.2.165.126:44241
I20260928 22:02:54.491606 19911 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:54.491822 19911 heartbeater.cc:511] Master 127.2.165.126:44241 requested a full tablet report, sending...
I20260928 22:02:54.492446 19720 ts_manager.cc:194] Registered new tserver with Master: edb5ee2c92794da294c826024a01448b (127.2.165.65:39399)
I20260928 22:02:54.493512 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:46819
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 22:02:54.520821 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44241
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:54.605365 19916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:54.605628 19916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:54.605669 19916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:54.606853 19916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:54.606925 19916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:54.606968 19916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:02:54.608444 19916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:54.608759 19916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:54.609005 19916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:54.614531 19924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:54.614579 19922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:54.614780 19916 server_base.cc:1034] running on GCE node
W20260928 22:02:54.614812 19921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:54.615162 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:54.615432 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:54.616648 19916 hybrid_clock.cc:648] HybridClock initialized: now 1790632974616623 us; error 39 us; skew 500 ppm
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:54.619345 19916 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:02:54.620234 19916 webserver.cc:460] Webserver started at http://127.2.165.66:44091/ using document root <none> and password file <none>
I20260928 22:02:54.620509 19916 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:54.620589 19916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:54.620781 19916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:54.621964 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "20ad66766c04ed8e152694ec8a648121"
server_key_iv: "8a3ccca0e8f255549f3fbc3a515c1600"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.622394 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "20ad66766c04ed8e152694ec8a648121"
server_key_iv: "8a3ccca0e8f255549f3fbc3a515c1600"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.624176 19916 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 22:02:54.625221 19931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.625506 19916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:02:54.625620 19916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "20ad66766c04ed8e152694ec8a648121"
server_key_iv: "8a3ccca0e8f255549f3fbc3a515c1600"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.625710 19916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:54.637670 19916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:54.639837 19916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:54.640012 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:54.640268 19916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:54.640625 19916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:54.640681 19916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.640797 19916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:54.640828 19916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.646973 19916 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:44047
I20260928 22:02:54.647030 20044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:44047 every 8 connection(s)
I20260928 22:02:54.647424 19916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:54.656389 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 19916
I20260928 22:02:54.656543 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:02:54.656808 2709 external_mini_cluster.cc:1428] Setting key 0a874c5c462ec7a43f0cbec6a04eab0b
I20260928 22:02:54.657652 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.648245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:33875 (local address 127.2.165.126:44241)
0928 22:02:54.648316 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.648319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.648610 (+ 291us) server_negotiation.cc:407] Connection header received
0928 22:02:54.649188 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.649189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.649201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.649269 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:54.650241 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.650741 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.651836 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.652077 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.652142 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.654642 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:54.654644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:54.654644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:54.654651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:54.655889 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.656356 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.656358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.656358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.656434 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.656812 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.656813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.656814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.656978 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:54.657046 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.657438 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.657541 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:02:54.657787 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.648240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:44241 (local address 127.2.165.66:33875)
0928 22:02:54.648476 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:54.648517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:54.649080 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:54.649328 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:54.649331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:54.649932 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:54.650136 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.650141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.650868 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.650869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.651744 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.651747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.652160 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.652161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.652264 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.653017 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:54.653026 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:54.654489 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:54.655992 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:54.655998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:54.656000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:54.656235 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:54.656519 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:54.656520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:54.656522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:54.656725 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:54.657037 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:54.657044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:54.657214 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 22:02:54.657321 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:02:54.657509 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260928 22:02:54.659363 20045 heartbeater.cc:347] Connected to a master server at 127.2.165.126:44241
I20260928 22:02:54.659526 20045 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:54.659745 20045 heartbeater.cc:511] Master 127.2.165.126:44241 requested a full tablet report, sending...
I20260928 22:02:54.660197 19720 ts_manager.cc:194] Registered new tserver with Master: a4c50b5b5e6e46ddbce728ed845a5ccf (127.2.165.66:44047)
I20260928 22:02:54.660607 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:33875
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 22:02:54.684582 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44241
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:54.770145 20050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:54.770363 20050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:54.770390 20050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:54.771544 20050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:54.771584 20050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:54.771611 20050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:02:54.773149 20050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:54.773365 20050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:54.773566 20050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:54.778975 20058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:54.779012 20055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:54.778970 20056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:54.779620 20050 server_base.cc:1034] running on GCE node
I20260928 22:02:54.779843 20050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:54.780090 20050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:54.781268 20050 hybrid_clock.cc:648] HybridClock initialized: now 1790632974781244 us; error 36 us; skew 500 ppm
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:54.784006 20050 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:02:54.784603 20050 webserver.cc:460] Webserver started at http://127.2.165.67:46499/ using document root <none> and password file <none>
I20260928 22:02:54.784929 20050 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:54.785001 20050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:54.785159 20050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:54.786247 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "ba9814fac209469dbb7789731566f6c2"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "8b5e9855360d87238b83d6d606184771"
server_key_iv: "978bc175bf48f087c8de172e7c531aea"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.786626 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "ba9814fac209469dbb7789731566f6c2"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "8b5e9855360d87238b83d6d606184771"
server_key_iv: "978bc175bf48f087c8de172e7c531aea"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.788100 20050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:02:54.789067 20065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.789331 20050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:54.789464 20050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "ba9814fac209469dbb7789731566f6c2"
format_stamp: "Formatted at 2026-09-28 22:02:54 on dist-test-slave-3t36"
server_key: "8b5e9855360d87238b83d6d606184771"
server_key_iv: "978bc175bf48f087c8de172e7c531aea"
server_key_version: "encryptionkey@0"
I20260928 22:02:54.789570 20050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:54.805868 20050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:54.808378 20050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:54.808586 20050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:54.808948 20050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:54.809312 20050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:54.809369 20050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.809417 20050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:54.809460 20050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:54.816042 20050 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:43917
I20260928 22:02:54.816121 20178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:43917 every 8 connection(s)
I20260928 22:02:54.816535 20050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:02:54.819933 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20050
I20260928 22:02:54.820040 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:02:54.820245 2709 external_mini_cluster.cc:1428] Setting key a174b27f1c27ad09a1a9fcfc2c326d5b
Sep 28 22:02:54 dist-test-slave-3t36 krb5kdc[19675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:54.827764 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.817531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:46873 (local address 127.2.165.126:44241)
0928 22:02:54.817607 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.817610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.817841 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:02:54.818454 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.818456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.818466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.818536 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:54.819703 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.820342 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.821959 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.822153 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.822217 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.824718 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:54.824720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:54.824720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:54.824727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:54.826278 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.826741 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.826743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.826743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.826803 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.827116 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.827117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.827118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.827239 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:54.827294 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.827619 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.827693 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:02:54.827965 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.817447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:44241 (local address 127.2.165.67:46873)
0928 22:02:54.817682 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:54.817730 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:54.818290 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:54.818680 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:54.818685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:54.819358 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:54.819588 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.819594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.820533 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.820535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.821855 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.821858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.822234 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.822234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.822341 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.822923 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:54.822932 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:54.824580 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:54.826372 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:54.826379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:54.826381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:54.826653 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:54.826899 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:54.826900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:54.826901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:54.827049 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:54.827317 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:54.827324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:54.827465 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 22:02:54.827580 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 22:02:54.827726 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260928 22:02:54.828606 20179 heartbeater.cc:347] Connected to a master server at 127.2.165.126:44241
I20260928 22:02:54.828756 20179 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:54.828943 20179 heartbeater.cc:511] Master 127.2.165.126:44241 requested a full tablet report, sending...
I20260928 22:02:54.829344 19720 ts_manager.cc:194] Registered new tserver with Master: ba9814fac209469dbb7789731566f6c2 (127.2.165.67:43917)
I20260928 22:02:54.829797 19720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:46873
I20260928 22:02:54.832332 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:02:54.840821 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.833924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.2.165.126:44241)
0928 22:02:54.834011 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.834013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.834039 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:02:54.834078 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.834079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.834088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.834130 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:54.834693 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.835191 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.836028 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.836218 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.836286 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.838008 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:54.838009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:54.838010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:54.838017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:54.839646 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.840019 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.840020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.840020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.840074 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:54.840266 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:54.840267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:54.840267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:54.840417 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:54.840491 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.840622 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.840744 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 22:02:54.842712 19720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:02:54.844473 19720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:02:54.851763 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.847245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.2.165.66:44047)
0928 22:02:54.847844 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.847849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.847861 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:02:54.847904 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.847906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.847923 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.848001 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:54.848260 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.849083 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.850431 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.851244 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.851323 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.851392 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.851627 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.851672 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":108,"threads_started":1}
I20260928 22:02:54.851754 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.847114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:44047 (local address 127.0.0.1:60480)
0928 22:02:54.847466 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:54.847516 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:54.847589 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:54.848090 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:54.848091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:54.848093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:54.848171 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.848173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.849193 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.849194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.850333 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.850336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.851441 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.851442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.851508 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.851521 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:02:54.851608 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 22:02:54.851672 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":41,"thread_start_us":56,"threads_started":1}
I20260928 22:02:54.851787 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.847401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.2.165.67:43917)
0928 22:02:54.847620 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.847624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.847906 (+ 282us) server_negotiation.cc:407] Connection header received
0928 22:02:54.848040 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.848042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.848058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.848115 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:54.848313 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.849128 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.850579 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.851308 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.851373 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.851419 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.851640 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.851701 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260928 22:02:54.852485 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.847139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39399 (local address 127.0.0.1:55884)
0928 22:02:54.847706 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:54.847716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:54.847781 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:54.848083 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:54.848087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:54.848089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:54.848169 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.848170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.850492 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.850493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.851201 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.851202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.852211 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.852212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.852288 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.852292 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:54.852340 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:02:54.852391 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":94,"threads_started":1}
I20260928 22:02:54.852581 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.847143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:43917 (local address 127.0.0.1:59540)
0928 22:02:54.847850 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:54.847859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:54.847946 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:54.848216 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:54.848216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:54.848218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:54.848258 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.848260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.849259 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.849261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.850495 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:54.850497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.851388 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.851389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:54.851466 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.851517 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 22:02:54.852423 (+ 906us) client_negotiation.cc:240] Negotiation successful
0928 22:02:54.852463 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":31,"thread_start_us":95,"threads_started":1}
I20260928 22:02:54.852558 19979 tablet_service.cc:1467] Processing CreateTablet for tablet dd2e4b1018d4458abb57c4da5c213198 (DEFAULT_TABLE table=test-table [id=58e6451b4a7d468ea63a6643eade4a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:54.852936 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:54.847624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.2.165.65:39399)
0928 22:02:54.847806 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:54.847810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:54.847821 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:02:54.847864 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:54.847866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:54.847883 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:54.847971 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:54.849196 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.850375 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.851291 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:54.852055 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:54.852123 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:54.852173 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:54.852783 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 22:02:54.852842 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260928 22:02:54.853051 19979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2e4b1018d4458abb57c4da5c213198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:54.853379 20113 tablet_service.cc:1467] Processing CreateTablet for tablet dd2e4b1018d4458abb57c4da5c213198 (DEFAULT_TABLE table=test-table [id=58e6451b4a7d468ea63a6643eade4a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:54.853569 19845 tablet_service.cc:1467] Processing CreateTablet for tablet dd2e4b1018d4458abb57c4da5c213198 (DEFAULT_TABLE table=test-table [id=58e6451b4a7d468ea63a6643eade4a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:54.853708 20113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2e4b1018d4458abb57c4da5c213198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:54.853914 19845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2e4b1018d4458abb57c4da5c213198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:54.856597 20196 tablet_bootstrap.cc:492] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Bootstrap starting.
I20260928 22:02:54.857111 20197 tablet_bootstrap.cc:492] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Bootstrap starting.
I20260928 22:02:54.857448 20196 tablet_bootstrap.cc:654] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:54.857758 20198 tablet_bootstrap.cc:492] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b: Bootstrap starting.
I20260928 22:02:54.857791 20197 tablet_bootstrap.cc:654] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:54.858052 20196 log.cc:824] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:54.858333 20197 log.cc:824] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:54.858409 20198 tablet_bootstrap.cc:654] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:54.858966 20196 tablet_bootstrap.cc:492] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: No bootstrap required, opened a new log
I20260928 22:02:54.859005 20198 log.cc:824] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:54.859047 20196 ts_tablet_manager.cc:1397] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:02:54.859091 20197 tablet_bootstrap.cc:492] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: No bootstrap required, opened a new log
I20260928 22:02:54.859162 20197 ts_tablet_manager.cc:1397] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:02:54.859787 20198 tablet_bootstrap.cc:492] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b: No bootstrap required, opened a new log
I20260928 22:02:54.859862 20198 ts_tablet_manager.cc:1397] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:02:54.860774 20196 raft_consensus.cc:348] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:54.860951 20196 raft_consensus.cc:374] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:54.860996 20196 raft_consensus.cc:729] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9814fac209469dbb7789731566f6c2, State: Initialized, Role: FOLLOWER
I20260928 22:02:54.861135 20196 consensus_queue.cc:260] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:54.861239 20197 raft_consensus.cc:348] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:54.861420 20197 raft_consensus.cc:374] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:54.861446 20179 heartbeater.cc:503] Master 127.2.165.126:44241 was elected leader, sending a full tablet report...
I20260928 22:02:54.861477 20197 raft_consensus.cc:729] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c50b5b5e6e46ddbce728ed845a5ccf, State: Initialized, Role: FOLLOWER
I20260928 22:02:54.861598 20197 consensus_queue.cc:260] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:54.861752 20196 ts_tablet_manager.cc:1428] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:02:54.861874 20197 ts_tablet_manager.cc:1428] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:02:54.861997 20045 heartbeater.cc:503] Master 127.2.165.126:44241 was elected leader, sending a full tablet report...
I20260928 22:02:54.861937 20198 raft_consensus.cc:348] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:54.862110 20198 raft_consensus.cc:374] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:54.862151 20198 raft_consensus.cc:729] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb5ee2c92794da294c826024a01448b, State: Initialized, Role: FOLLOWER
I20260928 22:02:54.862267 20198 consensus_queue.cc:260] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:54.862548 20198 ts_tablet_manager.cc:1428] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:02:54.862558 19911 heartbeater.cc:503] Master 127.2.165.126:44241 was elected leader, sending a full tablet report...
W20260928 22:02:54.898038 20046 tablet.cc:2369] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:02:54.980778 19912 tablet.cc:2369] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:02:55.000015 20202 raft_consensus.cc:482] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:55.000178 20202 raft_consensus.cc:504] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:55.000698 20202 leader_election.cc:288] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edb5ee2c92794da294c826024a01448b (127.2.165.65:39399), a4c50b5b5e6e46ddbce728ed845a5ccf (127.2.165.66:44047)
I20260928 22:02:55.003957 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.000878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39399 (local address 127.2.165.67:47253)
0928 22:02:55.000939 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.000951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.000991 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.001134 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.001135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.001136 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:55.001211 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.001213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.001980 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.001981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.002955 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.002958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.003612 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.003613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.003698 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.003702 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.003768 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.003814 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:02:55.004005 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.000919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:47253 (local address 127.2.165.65:39399)
0928 22:02:55.000967 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.000969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.000997 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:02:55.001064 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.001065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.001075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.001130 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:55.001321 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.001885 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.003073 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.003530 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.003587 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.003595 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.003874 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.003927 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:02:55.004204 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.000987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:34153 (local address 127.2.165.66:44047)
0928 22:02:55.001037 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.001039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.001264 (+ 225us) server_negotiation.cc:407] Connection header received
0928 22:02:55.001314 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.001316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.001325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.001375 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:55.001569 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.002162 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.003287 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.003808 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.003875 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.003883 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.004090 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.004142 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:02:55.004459 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.000953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:44047 (local address 127.2.165.67:34153)
0928 22:02:55.001173 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.001184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.001250 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.001391 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.001392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.001393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:55.001459 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.001460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.002268 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.002269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.003189 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.003192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.003895 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.003896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.003977 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.003982 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.004346 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.004393 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":170,"threads_started":1}
I20260928 22:02:55.004415 19865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2e4b1018d4458abb57c4da5c213198" candidate_uuid: "ba9814fac209469dbb7789731566f6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb5ee2c92794da294c826024a01448b" is_pre_election: true
I20260928 22:02:55.004590 19865 raft_consensus.cc:2454] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9814fac209469dbb7789731566f6c2 in term 0.
I20260928 22:02:55.004858 19999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2e4b1018d4458abb57c4da5c213198" candidate_uuid: "ba9814fac209469dbb7789731566f6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" is_pre_election: true
I20260928 22:02:55.004917 20067 leader_election.cc:302] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba9814fac209469dbb7789731566f6c2, edb5ee2c92794da294c826024a01448b; no voters:
I20260928 22:02:55.005017 19999 raft_consensus.cc:2454] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9814fac209469dbb7789731566f6c2 in term 0.
I20260928 22:02:55.005136 20202 raft_consensus.cc:2781] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:02:55.005215 20202 raft_consensus.cc:482] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:02:55.005241 20202 raft_consensus.cc:3037] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:55.005960 20202 raft_consensus.cc:504] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:55.006124 20202 leader_election.cc:288] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [CANDIDATE]: Term 1 election: Requested vote from peers edb5ee2c92794da294c826024a01448b (127.2.165.65:39399), a4c50b5b5e6e46ddbce728ed845a5ccf (127.2.165.66:44047)
I20260928 22:02:55.006296 19865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2e4b1018d4458abb57c4da5c213198" candidate_uuid: "ba9814fac209469dbb7789731566f6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb5ee2c92794da294c826024a01448b"
I20260928 22:02:55.006348 19999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2e4b1018d4458abb57c4da5c213198" candidate_uuid: "ba9814fac209469dbb7789731566f6c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf"
I20260928 22:02:55.006379 19865 raft_consensus.cc:3037] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:55.006424 19999 raft_consensus.cc:3037] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:55.007287 19865 raft_consensus.cc:2454] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9814fac209469dbb7789731566f6c2 in term 1.
I20260928 22:02:55.007287 19999 raft_consensus.cc:2454] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9814fac209469dbb7789731566f6c2 in term 1.
I20260928 22:02:55.007509 20068 leader_election.cc:302] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4c50b5b5e6e46ddbce728ed845a5ccf, ba9814fac209469dbb7789731566f6c2; no voters:
I20260928 22:02:55.007696 20202 raft_consensus.cc:2781] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:55.007876 20202 raft_consensus.cc:686] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 1 LEADER]: Becoming Leader. State: Replica: ba9814fac209469dbb7789731566f6c2, State: Running, Role: LEADER
I20260928 22:02:55.007989 20202 consensus_queue.cc:237] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } }
I20260928 22:02:55.008890 19718 catalog_manager.cc:5495] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9814fac209469dbb7789731566f6c2 (127.2.165.67). New cstate: current_term: 1 leader_uuid: "ba9814fac209469dbb7789731566f6c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba9814fac209469dbb7789731566f6c2" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43917 } health_report { overall_health: HEALTHY } } }
I20260928 22:02:55.035444 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.032843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.2.165.67:43917)
0928 22:02:55.032932 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.032935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.032944 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:02:55.032994 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.032995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.033015 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.033132 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:02:55.033277 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.033760 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.034603 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.034802 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.034884 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.035071 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:02:55.035177 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:02:55.035281 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.035300 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.035323 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 22:02:55.037457 19999 raft_consensus.cc:1261] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 1 FOLLOWER]: Refusing update from remote peer ba9814fac209469dbb7789731566f6c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:55.037498 19865 raft_consensus.cc:1261] T dd2e4b1018d4458abb57c4da5c213198 P edb5ee2c92794da294c826024a01448b [term 1 FOLLOWER]: Refusing update from remote peer ba9814fac209469dbb7789731566f6c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:55.037875 20202 consensus_queue.cc:1035] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c50b5b5e6e46ddbce728ed845a5ccf" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:55.038051 20206 consensus_queue.cc:1035] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb5ee2c92794da294c826024a01448b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 39399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:55.040661 20209 mvcc.cc:205] Tried to move back new op lower bound from 7334432665750642688 to 7334432665633488896. Current Snapshot: MvccSnapshot[applied={T|T < 7334432665750642688}]
I20260928 22:02:55.041404 20211 mvcc.cc:205] Tried to move back new op lower bound from 7334432665750642688 to 7334432665633488896. Current Snapshot: MvccSnapshot[applied={T|T < 7334432665750642688}]
I20260928 22:02:55.043320 19719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:55.043431 19719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:55.044261 19719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b19ecc65c1d4cba9bfd498e5f3943c9: Sending DeleteTablet for 3 replicas of tablet dd2e4b1018d4458abb57c4da5c213198
I20260928 22:02:55.044641 20113 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2e4b1018d4458abb57c4da5c213198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:55 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:59540
I20260928 22:02:55.044664 19845 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2e4b1018d4458abb57c4da5c213198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:55 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:55884
I20260928 22:02:55.044885 20218 tablet_replica.cc:330] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: stopping tablet replica
I20260928 22:02:55.044917 19979 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2e4b1018d4458abb57c4da5c213198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:55 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:60480
I20260928 22:02:55.044975 20218 raft_consensus.cc:2229] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:02:55.045069 20218 raft_consensus.cc:2258] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:55.045385 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 19782
I20260928 22:02:55.045513 20218 ts_tablet_manager.cc:1905] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:55.047374 20218 ts_tablet_manager.cc:1918] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:55.047441 20218 log.cc:1196] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/dd2e4b1018d4458abb57c4da5c213198
I20260928 22:02:55.047721 20218 ts_tablet_manager.cc:1939] T dd2e4b1018d4458abb57c4da5c213198 P ba9814fac209469dbb7789731566f6c2: Deleting consensus metadata
I20260928 22:02:55.048137 19707 catalog_manager.cc:4841] TS ba9814fac209469dbb7789731566f6c2 (127.2.165.67:43917): tablet dd2e4b1018d4458abb57c4da5c213198 (table test-table [id=58e6451b4a7d468ea63a6643eade4a24]) successfully deleted
I20260928 22:02:55.048820 20220 tablet_replica.cc:330] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: stopping tablet replica
I20260928 22:02:55.048951 20220 raft_consensus.cc:2229] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:55.049072 20220 raft_consensus.cc:2258] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:55.049541 20220 ts_tablet_manager.cc:1905] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:55.051136 20220 ts_tablet_manager.cc:1918] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:55.051218 20220 log.cc:1196] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/dd2e4b1018d4458abb57c4da5c213198
I20260928 22:02:55.051496 20220 ts_tablet_manager.cc:1939] T dd2e4b1018d4458abb57c4da5c213198 P a4c50b5b5e6e46ddbce728ed845a5ccf: Deleting consensus metadata
W20260928 22:02:55.051832 19705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:39399 (error 108)
I20260928 22:02:55.052088 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 19916
I20260928 22:02:55.052139 19706 catalog_manager.cc:4841] TS a4c50b5b5e6e46ddbce728ed845a5ccf (127.2.165.66:44047): tablet dd2e4b1018d4458abb57c4da5c213198 (table test-table [id=58e6451b4a7d468ea63a6643eade4a24]) successfully deleted
I20260928 22:02:55.053848 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.053689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:39399 (local address 127.0.0.1:55888)
0928 22:02:55.053761 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.053804 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 22:02:55.053996 19705 catalog_manager.cc:4584] TS edb5ee2c92794da294c826024a01448b (127.2.165.65:39399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd2e4b1018d4458abb57c4da5c213198: Network error: Client connection negotiation failed: client connection to 127.2.165.65:39399: connect: Connection refused (error 111)
I20260928 22:02:55.057497 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20050
I20260928 22:02:55.061920 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 19688
2026-09-28T22:02:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T22:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:02:55Z Disabled control of system clock
I20260928 22:02:55.090428 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45311
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:02:55.173856 20225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:55.174088 20225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:55.174111 20225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:55.175331 20225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:02:55.175376 20225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:55.175390 20225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:55.175402 20225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:02:55.175415 20225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:02:55.176743 20225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:55.176939 20225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:55.177181 20225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:55.182731 20233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:55.182717 20230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:55.182761 20225 server_base.cc:1034] running on GCE node
W20260928 22:02:55.182717 20231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:55.183362 20225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:55.183652 20225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:55.184756 20225 hybrid_clock.cc:648] HybridClock initialized: now 1790632975184730 us; error 38 us; skew 500 ppm
I20260928 22:02:55.186228 20225 webserver.cc:460] Webserver started at http://127.2.165.126:46189/ using document root <none> and password file <none>
I20260928 22:02:55.186465 20225 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:55.186512 20225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:55.186607 20225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:55.187681 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "bb59faa50f0446d4a7908474170184a3"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "3a1a05934b19999d66c0e6de8b8a3b84"
server_key_iv: "559aba5809c7e8e044a5af55ce8945cd"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.188007 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "bb59faa50f0446d4a7908474170184a3"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "3a1a05934b19999d66c0e6de8b8a3b84"
server_key_iv: "559aba5809c7e8e044a5af55ce8945cd"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.189391 20225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:02:55.190120 20239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.190321 20225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:55.190418 20225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "bb59faa50f0446d4a7908474170184a3"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "3a1a05934b19999d66c0e6de8b8a3b84"
server_key_iv: "559aba5809c7e8e044a5af55ce8945cd"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.190510 20225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:55.205755 20225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:55.206138 20225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:55.206311 20225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:55.210561 20225 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45311
I20260928 22:02:55.210598 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45311 every 8 connection(s)
I20260928 22:02:55.211000 20225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:02:55.211993 20292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:55.214440 20292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3: Bootstrap starting.
I20260928 22:02:55.215060 20292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:55.215332 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20225
I20260928 22:02:55.215459 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:02:55.215452 20292 log.cc:824] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:55.215672 2709 external_mini_cluster.cc:1428] Setting key 10302fb96133b3b74ceaccf4a1a011ae
I20260928 22:02:55.216390 20292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3: No bootstrap required, opened a new log
I20260928 22:02:55.218152 20292 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb59faa50f0446d4a7908474170184a3" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45311 } }
I20260928 22:02:55.218319 20292 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:55.218365 20292 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb59faa50f0446d4a7908474170184a3, State: Initialized, Role: FOLLOWER
I20260928 22:02:55.218515 20292 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb59faa50f0446d4a7908474170184a3" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45311 } }
I20260928 22:02:55.218607 20292 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:02:55.218653 20292 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:02:55.218727 20292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:55.219318 20292 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb59faa50f0446d4a7908474170184a3" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45311 } }
I20260928 22:02:55.219439 20292 leader_election.cc:302] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb59faa50f0446d4a7908474170184a3; no voters:
I20260928 22:02:55.219650 20292 leader_election.cc:288] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:02:55.219682 20297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:55.219785 20297 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [term 1 LEADER]: Becoming Leader. State: Replica: bb59faa50f0446d4a7908474170184a3, State: Running, Role: LEADER
I20260928 22:02:55.219868 20297 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb59faa50f0446d4a7908474170184a3" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45311 } }
I20260928 22:02:55.219929 20292 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:02:55.220227 20297 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb59faa50f0446d4a7908474170184a3. Latest consensus state: current_term: 1 leader_uuid: "bb59faa50f0446d4a7908474170184a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb59faa50f0446d4a7908474170184a3" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45311 } } }
I20260928 22:02:55.220304 20297 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:55.220237 20298 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb59faa50f0446d4a7908474170184a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb59faa50f0446d4a7908474170184a3" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45311 } } }
I20260928 22:02:55.220503 20298 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:55.220849 20302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:02:55.221525 20302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:02:55.223614 20302 catalog_manager.cc:1347] Generated new cluster ID: 21dda532f15e4a5dbd0fc31069c966a9
I20260928 22:02:55.223667 20302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:02:55.223882 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.216275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.2.165.126:45311)
0928 22:02:55.216548 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.216557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.216596 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:02:55.218147 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.218157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.218228 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.218404 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:55.218611 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.220410 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.221670 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.222207 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.222405 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.223115 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:55.223120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:55.223120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:55.223335 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:55.223370 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:55.223382 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:55.223465 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.223589 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.223684 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":121,"threads_started":1}
I20260928 22:02:55.242103 20302 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:02:55.242735 20302 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:02:55.248520 20302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3: Generated new TSK 0
I20260928 22:02:55.248790 20302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:02:55.255715 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45311
--builtin_ntp_servers=127.2.165.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:02:55.362571 20316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:55.362885 20316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:55.362923 20316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:55.364114 20316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:55.364184 20316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:55.364231 20316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:02:55.365688 20316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:55.365964 20316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:55.366277 20316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:55.371382 20322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:55.371750 20316 server_base.cc:1034] running on GCE node
W20260928 22:02:55.371528 20321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:55.371376 20324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:55.372056 20316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:55.372315 20316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:55.373497 20316 hybrid_clock.cc:648] HybridClock initialized: now 1790632975373475 us; error 31 us; skew 500 ppm
I20260928 22:02:55.374977 20316 webserver.cc:460] Webserver started at http://127.2.165.65:36949/ using document root <none> and password file <none>
I20260928 22:02:55.375252 20316 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:55.375319 20316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:55.375468 20316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:55.377033 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "cacd1422b06343df8e71424a301b0fb4"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "88620a0afecf2047aa73530a6ce32780"
server_key_iv: "cefc207ca8225c1b4e88772ace4091cb"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.377508 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "cacd1422b06343df8e71424a301b0fb4"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "88620a0afecf2047aa73530a6ce32780"
server_key_iv: "cefc207ca8225c1b4e88772ace4091cb"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.379196 20316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:02:55.380070 20330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.380280 20316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:55.380383 20316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "cacd1422b06343df8e71424a301b0fb4"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "88620a0afecf2047aa73530a6ce32780"
server_key_iv: "cefc207ca8225c1b4e88772ace4091cb"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.380460 20316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:55.402810 20316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:55.403167 20316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:55.403327 20316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:55.403580 20316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:55.403926 20316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:55.403960 20316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.403985 20316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:55.404030 20316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.409917 20316 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:37167
I20260928 22:02:55.410048 20443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:37167 every 8 connection(s)
I20260928 22:02:55.410375 20316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:02:55.412178 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20316
I20260928 22:02:55.412312 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:02:55.412554 2709 external_mini_cluster.cc:1428] Setting key a2482020d4e50a6d8059792046c90daa
I20260928 22:02:55.413774 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45311
--builtin_ntp_servers=127.2.165.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:02:55.416937 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.411213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:50091 (local address 127.2.165.126:45311)
0928 22:02:55.411308 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.411323 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.411546 (+ 223us) server_negotiation.cc:407] Connection header received
0928 22:02:55.412077 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.412079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.412096 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.412180 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:55.413202 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.413983 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.415159 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.415378 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.415443 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.416616 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:55.416617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:55.416618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:55.416623 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:55.416640 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:55.416645 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:55.416714 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.416818 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.416856 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 22:02:55.417178 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.411189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45311 (local address 127.2.165.65:50091)
0928 22:02:55.411413 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.411457 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.411949 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.412279 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.412284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.412295 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:55.413099 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.413106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.414080 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.414082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.415050 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.415053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.415468 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.415469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.415644 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.416420 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:02:55.416421 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:55.416452 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:02:55.416477 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:55.416724 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:55.416729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:55.416733 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.416823 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.416933 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
I20260928 22:02:55.417771 20444 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45311
I20260928 22:02:55.417887 20444 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:55.418081 20444 heartbeater.cc:511] Master 127.2.165.126:45311 requested a full tablet report, sending...
I20260928 22:02:55.418803 20256 ts_manager.cc:194] Registered new tserver with Master: cacd1422b06343df8e71424a301b0fb4 (127.2.165.65:37167)
I20260928 22:02:55.419890 20256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.65:50091
W20260928 22:02:55.501680 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:55.501916 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:55.501941 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:55.503168 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:55.503213 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:55.503240 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:02:55.504622 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:55.504920 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:55.505221 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:02:55.510720 20447 server_base.cc:1034] running on GCE node
W20260928 22:02:55.510692 20452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:55.510699 20455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:55.510699 20453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:55.511230 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:55.511514 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:55.512774 20447 hybrid_clock.cc:648] HybridClock initialized: now 1790632975512755 us; error 32 us; skew 500 ppm
I20260928 22:02:55.514266 20447 webserver.cc:460] Webserver started at http://127.2.165.66:37023/ using document root <none> and password file <none>
I20260928 22:02:55.514561 20447 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:55.514642 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:55.514801 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:55.515944 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "5f9203168b674c529dbc31c1e19ba924"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "43401fa6287a9357ad663de624f238d3"
server_key_iv: "2a386ac5e49714ec6c9ac75337b25e67"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.516520 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "5f9203168b674c529dbc31c1e19ba924"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "43401fa6287a9357ad663de624f238d3"
server_key_iv: "2a386ac5e49714ec6c9ac75337b25e67"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.518046 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:55.518890 20461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.519145 20447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:55.519251 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "5f9203168b674c529dbc31c1e19ba924"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "43401fa6287a9357ad663de624f238d3"
server_key_iv: "2a386ac5e49714ec6c9ac75337b25e67"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.519328 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:55.528919 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:55.529241 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:55.529392 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:55.529621 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:55.529949 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:55.530004 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.530035 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:55.530052 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.536026 20447 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:39865
I20260928 22:02:55.536078 20574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:39865 every 8 connection(s)
I20260928 22:02:55.536428 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:02:55.539251 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20447
I20260928 22:02:55.539361 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:02:55.539633 2709 external_mini_cluster.cc:1428] Setting key 696a358c0250b97d874c17cc0ed812f9
I20260928 22:02:55.540853 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45311
--builtin_ntp_servers=127.2.165.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:02:55.542686 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.537185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:58727 (local address 127.2.165.126:45311)
0928 22:02:55.537269 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.537272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.537528 (+ 256us) server_negotiation.cc:407] Connection header received
0928 22:02:55.538014 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.538016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.538026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.538072 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:55.538802 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.539540 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.540923 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.541242 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.541326 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.542275 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:55.542276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:55.542277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:55.542284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:55.542312 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:55.542318 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:55.542403 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.542575 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.542610 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:02:55.543015 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.537178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45311 (local address 127.2.165.66:58727)
0928 22:02:55.537396 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.537432 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.537920 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.538133 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.538137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.538145 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:55.538684 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.538689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.539649 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.539650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.540831 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.540833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.541334 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.541335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.541458 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.542092 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:02:55.542092 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:55.542125 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:02:55.542150 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:55.542434 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:55.542442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:55.542447 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.542559 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.542730 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":118,"threads_started":1}
I20260928 22:02:55.543679 20575 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45311
I20260928 22:02:55.543813 20575 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:55.544045 20575 heartbeater.cc:511] Master 127.2.165.126:45311 requested a full tablet report, sending...
I20260928 22:02:55.544605 20256 ts_manager.cc:194] Registered new tserver with Master: 5f9203168b674c529dbc31c1e19ba924 (127.2.165.66:39865)
I20260928 22:02:55.545058 20256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.66:58727
W20260928 22:02:55.629010 20578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:55.629238 20578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:55.629262 20578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:55.630492 20578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:55.630553 20578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:55.630581 20578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:02:55.632012 20578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:45311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:55.632292 20578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:55.632704 20578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:55.638120 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:55.638136 20583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:55.638118 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:55.638139 20578 server_base.cc:1034] running on GCE node
I20260928 22:02:55.638577 20578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:55.638867 20578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:55.640048 20578 hybrid_clock.cc:648] HybridClock initialized: now 1790632975640032 us; error 29 us; skew 500 ppm
I20260928 22:02:55.641396 20578 webserver.cc:460] Webserver started at http://127.2.165.67:37867/ using document root <none> and password file <none>
I20260928 22:02:55.641774 20578 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:55.641863 20578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:55.642030 20578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:55.643160 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "3664782357c74f0dbfde47e65c8934ef"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "bcd8edfd93c520da5a898dfc45900077"
server_key_iv: "d33299c7c34b3fc269708d57f44ff2e5"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.643538 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "3664782357c74f0dbfde47e65c8934ef"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "bcd8edfd93c520da5a898dfc45900077"
server_key_iv: "d33299c7c34b3fc269708d57f44ff2e5"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.644951 20578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:02:55.645861 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.646107 20578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:55.646219 20578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "3664782357c74f0dbfde47e65c8934ef"
format_stamp: "Formatted at 2026-09-28 22:02:55 on dist-test-slave-3t36"
server_key: "bcd8edfd93c520da5a898dfc45900077"
server_key_iv: "d33299c7c34b3fc269708d57f44ff2e5"
server_key_version: "encryptionkey@0"
I20260928 22:02:55.646309 20578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:55.675282 20578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:55.675645 20578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:55.675807 20578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:55.676097 20578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:55.676450 20578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:55.676506 20578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.676538 20578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:55.676555 20578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:55.682819 20578 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:43903
I20260928 22:02:55.682977 20705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:43903 every 8 connection(s)
I20260928 22:02:55.683311 20578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:02:55.687831 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20578
I20260928 22:02:55.687971 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:02:55.688164 2709 external_mini_cluster.cc:1428] Setting key 96f2c7d7b9ef0af070a3a7d66fba2a5d
I20260928 22:02:55.688851 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.684289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:51083 (local address 127.2.165.126:45311)
0928 22:02:55.684352 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.684354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.684606 (+ 252us) server_negotiation.cc:407] Connection header received
0928 22:02:55.685108 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.685109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.685120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.685180 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:55.685829 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.686332 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.687299 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.687486 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.687538 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.688527 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:55.688528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:55.688528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:55.688533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:55.688553 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:55.688557 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:55.688633 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.688733 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.688771 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:02:55.689097 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.684269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45311 (local address 127.2.165.67:51083)
0928 22:02:55.684453 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.684489 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.685006 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.685233 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.685236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.685244 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:55.685752 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.685756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.686416 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.686417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.687218 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.687221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.687577 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.687578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.687754 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.688363 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:02:55.688364 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:55.688408 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:02:55.688438 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:55.688633 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:55.688638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:55.688641 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.688735 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.688839 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260928 22:02:55.689569 20706 heartbeater.cc:347] Connected to a master server at 127.2.165.126:45311
I20260928 22:02:55.689657 20706 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:55.689846 20706 heartbeater.cc:511] Master 127.2.165.126:45311 requested a full tablet report, sending...
I20260928 22:02:55.690225 20256 ts_manager.cc:194] Registered new tserver with Master: 3664782357c74f0dbfde47e65c8934ef (127.2.165.67:43903)
I20260928 22:02:55.690565 20256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.67:51083
I20260928 22:02:55.700381 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:02:55.705241 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.702032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.2.165.126:45311)
0928 22:02:55.702104 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.702107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.702235 (+ 128us) server_negotiation.cc:407] Connection header received
0928 22:02:55.702307 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.702308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.702323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.702408 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:55.702604 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.703127 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.703905 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.704106 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.704185 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.704949 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:55.704951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:55.704951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:55.704956 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:55.704970 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:55.704975 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:55.705052 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.705115 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.705145 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:02:55.706815 20256 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:02:55.708420 20256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:02:55.714941 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.710794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.2.165.66:39865)
0928 22:02:55.711067 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.711073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.711141 (+ 68us) server_negotiation.cc:407] Connection header received
0928 22:02:55.711292 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.711295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.711316 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.711394 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:55.711672 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.712863 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.713991 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.714550 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.714631 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.714732 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.714807 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.714847 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":87,"threads_started":1}
I20260928 22:02:55.714932 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.710703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:39865 (local address 127.0.0.1:43964)
0928 22:02:55.711014 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.711059 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.711175 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.711464 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.711465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.711467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:55.711580 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.711582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.712956 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.712957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.713902 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.713905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.714622 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.714623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.714701 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.714713 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.714792 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.714847 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":81,"threads_started":1}
I20260928 22:02:55.715215 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.710731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:37167 (local address 127.0.0.1:52126)
0928 22:02:55.711062 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.711075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.711142 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.711334 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.711337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.711339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:55.711426 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.711428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.712425 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.712426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.713717 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.713720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.714777 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.714778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.714845 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.714849 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.714895 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.714941 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":64,"threads_started":1}
I20260928 22:02:55.715370 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.710830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.2.165.65:37167)
0928 22:02:55.711150 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.711153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.711163 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:02:55.711231 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.711233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.711251 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.711321 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:55.711517 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.712329 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.713809 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.714572 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.714646 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.714736 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.715147 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.715190 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":79,"threads_started":1}
I20260928 22:02:55.715524 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.711064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.2.165.67:43903)
0928 22:02:55.711246 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.711251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.711418 (+ 167us) server_negotiation.cc:407] Connection header received
0928 22:02:55.711459 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.711461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.711474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.711540 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:55.711720 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.712834 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.714066 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.714775 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.714838 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.714915 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.715382 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.715427 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260928 22:02:55.715709 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.710938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:43903 (local address 127.0.0.1:45842)
0928 22:02:55.711302 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.711312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.711364 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.711609 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.711610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.711612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:55.711646 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.711648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.712929 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.712930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.714000 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.714002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.714946 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.714947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.715000 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.715003 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.715254 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.715642 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":26,"thread_start_us":71,"threads_started":1}
I20260928 22:02:55.715772 20509 tablet_service.cc:1467] Processing CreateTablet for tablet 67c5c04de6f64834a1286d10eab3e75f (DEFAULT_TABLE table=test-table [id=25824215afd14ae6a599842bde238cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:55.716192 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c5c04de6f64834a1286d10eab3e75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:55.716447 20640 tablet_service.cc:1467] Processing CreateTablet for tablet 67c5c04de6f64834a1286d10eab3e75f (DEFAULT_TABLE table=test-table [id=25824215afd14ae6a599842bde238cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:55.716801 20640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c5c04de6f64834a1286d10eab3e75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:55.719894 20720 tablet_bootstrap.cc:492] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: Bootstrap starting.
I20260928 22:02:55.720276 20721 tablet_bootstrap.cc:492] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Bootstrap starting.
I20260928 22:02:55.720552 20378 tablet_service.cc:1467] Processing CreateTablet for tablet 67c5c04de6f64834a1286d10eab3e75f (DEFAULT_TABLE table=test-table [id=25824215afd14ae6a599842bde238cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:55.720883 20720 tablet_bootstrap.cc:654] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:55.720973 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c5c04de6f64834a1286d10eab3e75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:55.720996 20721 tablet_bootstrap.cc:654] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:55.721259 20720 log.cc:824] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:55.721299 20721 log.cc:824] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:55.722175 20720 tablet_bootstrap.cc:492] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: No bootstrap required, opened a new log
I20260928 22:02:55.722263 20720 ts_tablet_manager.cc:1397] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:02:55.724017 20724 tablet_bootstrap.cc:492] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4: Bootstrap starting.
I20260928 22:02:55.724462 20721 tablet_bootstrap.cc:492] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: No bootstrap required, opened a new log
I20260928 22:02:55.724553 20721 ts_tablet_manager.cc:1397] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 22:02:55.724771 20724 tablet_bootstrap.cc:654] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:55.725142 20724 log.cc:824] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:55.725011 20720 raft_consensus.cc:348] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.725210 20720 raft_consensus.cc:374] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:55.725250 20720 raft_consensus.cc:729] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f9203168b674c529dbc31c1e19ba924, State: Initialized, Role: FOLLOWER
I20260928 22:02:55.725394 20720 consensus_queue.cc:260] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.725688 20720 ts_tablet_manager.cc:1428] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:02:55.725836 20575 heartbeater.cc:503] Master 127.2.165.126:45311 was elected leader, sending a full tablet report...
I20260928 22:02:55.726048 20724 tablet_bootstrap.cc:492] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4: No bootstrap required, opened a new log
I20260928 22:02:55.726133 20724 ts_tablet_manager.cc:1397] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:02:55.727129 20721 raft_consensus.cc:348] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.727259 20721 raft_consensus.cc:374] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:55.727280 20721 raft_consensus.cc:729] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3664782357c74f0dbfde47e65c8934ef, State: Initialized, Role: FOLLOWER
I20260928 22:02:55.727401 20721 consensus_queue.cc:260] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.727679 20721 ts_tablet_manager.cc:1428] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:02:55.727715 20706 heartbeater.cc:503] Master 127.2.165.126:45311 was elected leader, sending a full tablet report...
I20260928 22:02:55.729053 20724 raft_consensus.cc:348] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.729213 20724 raft_consensus.cc:374] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:55.729261 20724 raft_consensus.cc:729] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacd1422b06343df8e71424a301b0fb4, State: Initialized, Role: FOLLOWER
I20260928 22:02:55.729385 20724 consensus_queue.cc:260] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.729655 20724 ts_tablet_manager.cc:1428] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 22:02:55.729745 20444 heartbeater.cc:503] Master 127.2.165.126:45311 was elected leader, sending a full tablet report...
W20260928 22:02:55.787101 20576 tablet.cc:2369] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:02:55.800915 20726 raft_consensus.cc:482] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:55.801057 20726 raft_consensus.cc:504] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.801482 20726 leader_election.cc:288] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3664782357c74f0dbfde47e65c8934ef (127.2.165.67:43903), cacd1422b06343df8e71424a301b0fb4 (127.2.165.65:37167)
I20260928 22:02:55.805248 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.801929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:40197 (local address 127.2.165.65:37167)
0928 22:02:55.801976 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.801978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.801990 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:02:55.802021 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.802022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.802034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.802110 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:55.802332 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.803123 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.804399 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.804845 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.804953 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.804963 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.805119 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.805166 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:02:55.805248 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.801958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:33911 (local address 127.2.165.67:43903)
0928 22:02:55.802028 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.802030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.802165 (+ 135us) server_negotiation.cc:407] Connection header received
0928 22:02:55.802204 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.802204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.802211 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.802258 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:55.802421 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.803152 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.804386 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.804860 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.804950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.804960 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.805120 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.805164 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:02:55.805270 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.801735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:37167 (local address 127.2.165.66:40197)
0928 22:02:55.801831 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.801844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.801920 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.802123 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.802125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.802126 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:55.802213 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.802215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.803248 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.803249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.804284 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.804288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.804953 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.804954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.805036 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.805040 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.805110 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.805157 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":86}
I20260928 22:02:55.805292 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.801831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:43903 (local address 127.2.165.66:33911)
0928 22:02:55.802083 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.802094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:55.802150 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:55.802288 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:55.802289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:55.802291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:55.802353 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.802354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.803247 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.803249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.804278 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:55.804281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.804950 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.804951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:55.805039 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.805042 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:02:55.805111 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:02:55.805154 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":182,"threads_started":1}
I20260928 22:02:55.805666 20660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c5c04de6f64834a1286d10eab3e75f" candidate_uuid: "5f9203168b674c529dbc31c1e19ba924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3664782357c74f0dbfde47e65c8934ef" is_pre_election: true
I20260928 22:02:55.805673 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c5c04de6f64834a1286d10eab3e75f" candidate_uuid: "5f9203168b674c529dbc31c1e19ba924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacd1422b06343df8e71424a301b0fb4" is_pre_election: true
I20260928 22:02:55.805846 20660 raft_consensus.cc:2454] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9203168b674c529dbc31c1e19ba924 in term 0.
I20260928 22:02:55.805846 20398 raft_consensus.cc:2454] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f9203168b674c529dbc31c1e19ba924 in term 0.
I20260928 22:02:55.806111 20465 leader_election.cc:302] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3664782357c74f0dbfde47e65c8934ef, 5f9203168b674c529dbc31c1e19ba924; no voters:
I20260928 22:02:55.806268 20726 raft_consensus.cc:2781] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:02:55.806326 20726 raft_consensus.cc:482] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:02:55.806352 20726 raft_consensus.cc:3037] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:55.807062 20726 raft_consensus.cc:504] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.807246 20726 leader_election.cc:288] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [CANDIDATE]: Term 1 election: Requested vote from peers 3664782357c74f0dbfde47e65c8934ef (127.2.165.67:43903), cacd1422b06343df8e71424a301b0fb4 (127.2.165.65:37167)
I20260928 22:02:55.807387 20660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c5c04de6f64834a1286d10eab3e75f" candidate_uuid: "5f9203168b674c529dbc31c1e19ba924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3664782357c74f0dbfde47e65c8934ef"
I20260928 22:02:55.807472 20660 raft_consensus.cc:3037] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:55.808166 20660 raft_consensus.cc:2454] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9203168b674c529dbc31c1e19ba924 in term 1.
I20260928 22:02:55.808313 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c5c04de6f64834a1286d10eab3e75f" candidate_uuid: "5f9203168b674c529dbc31c1e19ba924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacd1422b06343df8e71424a301b0fb4"
I20260928 22:02:55.808375 20465 leader_election.cc:302] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3664782357c74f0dbfde47e65c8934ef, 5f9203168b674c529dbc31c1e19ba924; no voters:
I20260928 22:02:55.808408 20398 raft_consensus.cc:3037] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:55.808490 20726 raft_consensus.cc:2781] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:55.808692 20726 raft_consensus.cc:686] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 1 LEADER]: Becoming Leader. State: Replica: 5f9203168b674c529dbc31c1e19ba924, State: Running, Role: LEADER
I20260928 22:02:55.808807 20726 consensus_queue.cc:237] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } }
I20260928 22:02:55.809417 20398 raft_consensus.cc:2454] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9203168b674c529dbc31c1e19ba924 in term 1.
I20260928 22:02:55.809835 20256 catalog_manager.cc:5495] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f9203168b674c529dbc31c1e19ba924 (127.2.165.66). New cstate: current_term: 1 leader_uuid: "5f9203168b674c529dbc31c1e19ba924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9203168b674c529dbc31c1e19ba924" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 39865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 } health_report { overall_health: UNKNOWN } } }
I20260928 22:02:55.831913 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.829359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.2.165.66:39865)
0928 22:02:55.829421 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:55.829423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:55.829523 (+ 100us) server_negotiation.cc:407] Connection header received
0928 22:02:55.829561 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:55.829562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:55.829582 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:55.829649 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:02:55.829862 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.830342 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.831106 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:55.831299 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:55.831365 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:55.831538 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:02:55.831652 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:02:55.831718 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:55.831756 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:02:55.831776 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:02:55.834161 20660 raft_consensus.cc:1261] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 1 FOLLOWER]: Refusing update from remote peer 5f9203168b674c529dbc31c1e19ba924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:55.834161 20398 raft_consensus.cc:1261] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 1 FOLLOWER]: Refusing update from remote peer 5f9203168b674c529dbc31c1e19ba924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:55.834569 20726 consensus_queue.cc:1035] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3664782357c74f0dbfde47e65c8934ef" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:55.834707 20730 consensus_queue.cc:1035] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cacd1422b06343df8e71424a301b0fb4" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:55.837236 20733 mvcc.cc:205] Tried to move back new op lower bound from 7334432669012922368 to 7334432668913512448. Current Snapshot: MvccSnapshot[applied={T|T < 7334432669012922368}]
I20260928 22:02:55.837728 20735 mvcc.cc:205] Tried to move back new op lower bound from 7334432669012922368 to 7334432668913512448. Current Snapshot: MvccSnapshot[applied={T|T < 7334432669012922368}]
I20260928 22:02:55.838060 20734 mvcc.cc:205] Tried to move back new op lower bound from 7334432669012922368 to 7334432668913512448. Current Snapshot: MvccSnapshot[applied={T|T < 7334432669012922368}]
I20260928 22:02:55.839488 20256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:55.839655 20256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:55.840451 20256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb59faa50f0446d4a7908474170184a3: Sending DeleteTablet for 3 replicas of tablet 67c5c04de6f64834a1286d10eab3e75f
I20260928 22:02:55.840832 20509 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c5c04de6f64834a1286d10eab3e75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:55 UTC) from {username='slave'} at 127.0.0.1:43964
I20260928 22:02:55.840876 20378 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c5c04de6f64834a1286d10eab3e75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:55 UTC) from {username='slave'} at 127.0.0.1:52126
I20260928 22:02:55.841116 20640 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c5c04de6f64834a1286d10eab3e75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:55 UTC) from {username='slave'} at 127.0.0.1:45842
I20260928 22:02:55.841334 20745 tablet_replica.cc:330] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: stopping tablet replica
I20260928 22:02:55.841389 20744 tablet_replica.cc:330] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4: stopping tablet replica
I20260928 22:02:55.841499 20744 raft_consensus.cc:2229] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:55.841528 20745 raft_consensus.cc:2229] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:55.841615 20744 raft_consensus.cc:2258] T 67c5c04de6f64834a1286d10eab3e75f P cacd1422b06343df8e71424a301b0fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:55.841632 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20316
I20260928 22:02:55.842413 20745 raft_consensus.cc:2258] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:55.843158 20745 ts_tablet_manager.cc:1905] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:55.844965 20745 ts_tablet_manager.cc:1918] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:55.845047 20745 log.cc:1196] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/67c5c04de6f64834a1286d10eab3e75f
I20260928 22:02:55.845323 20745 ts_tablet_manager.cc:1939] T 67c5c04de6f64834a1286d10eab3e75f P 3664782357c74f0dbfde47e65c8934ef: Deleting consensus metadata
I20260928 22:02:55.845803 20243 catalog_manager.cc:4841] TS 3664782357c74f0dbfde47e65c8934ef (127.2.165.67:43903): tablet 67c5c04de6f64834a1286d10eab3e75f (table test-table [id=25824215afd14ae6a599842bde238cf4]) successfully deleted
W20260928 22:02:55.847071 20240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:37167 (error 108)
I20260928 22:02:55.847601 20743 tablet_replica.cc:330] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: stopping tablet replica
I20260928 22:02:55.847607 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.847450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:37167 (local address 127.0.0.1:52132)
0928 22:02:55.847515 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.847563 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:02:55.847728 20743 raft_consensus.cc:2229] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:02:55.847836 20743 raft_consensus.cc:2258] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 22:02:55.848057 20240 catalog_manager.cc:4584] TS cacd1422b06343df8e71424a301b0fb4 (127.2.165.65:37167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67c5c04de6f64834a1286d10eab3e75f: Network error: Client connection negotiation failed: client connection to 127.2.165.65:37167: connect: Connection refused (error 111)
I20260928 22:02:55.848280 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20447
I20260928 22:02:55.848280 20743 ts_tablet_manager.cc:1905] T 67c5c04de6f64834a1286d10eab3e75f P 5f9203168b674c529dbc31c1e19ba924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:55.855149 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:55.854930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:39865 (local address 127.0.0.1:43994)
0928 22:02:55.854994 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:55.855078 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.66:39865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:02:55.855211 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20578
I20260928 22:02:55.859943 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20225
2026-09-28T22:02:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:02:55 dist-test-slave-3t36 krb5kdc[20749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:02:55 dist-test-slave-3t36 krb5kdc[20749](info): set up 2 sockets
Sep 28 22:02:55 dist-test-slave-3t36 krb5kdc[20749](info): commencing operation
krb5kdc: starting...
W20260928 22:02:56.928639 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:02:57.056463 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:40387
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:57.138567 20762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:57.138844 20762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:57.138880 20762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:57.140085 20762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:02:57.140173 20762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:57.140197 20762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:57.140218 20762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:02:57.140250 20762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:02:57.141891 20762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:57.142216 20762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:57.142500 20762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:57.147656 20767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:57.147656 20768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:57.147748 20770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:57.148263 20762 server_base.cc:1034] running on GCE node
I20260928 22:02:57.148473 20762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:57.148803 20762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:57.149974 20762 hybrid_clock.cc:648] HybridClock initialized: now 1790632977149960 us; error 33 us; skew 500 ppm
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:57.152899 20762 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:02:57.153395 20762 webserver.cc:460] Webserver started at http://127.2.165.126:45313/ using document root <none> and password file <none>
I20260928 22:02:57.153674 20762 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:57.153724 20762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:57.153859 20762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:57.154980 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "553897d47b6d4743b50fe5243609d900"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "d317623488c090ccffb62d6fdf0ef4c7"
server_key_iv: "8dec0afe1008d0cb5c18067ed6a16659"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.155364 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "553897d47b6d4743b50fe5243609d900"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "d317623488c090ccffb62d6fdf0ef4c7"
server_key_iv: "8dec0afe1008d0cb5c18067ed6a16659"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.156864 20762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:02:57.157796 20777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.158051 20762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:57.158177 20762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "553897d47b6d4743b50fe5243609d900"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "d317623488c090ccffb62d6fdf0ef4c7"
server_key_iv: "8dec0afe1008d0cb5c18067ed6a16659"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.158262 20762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:57.168345 20762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:57.170938 20762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:57.171119 20762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:57.175379 20762 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:40387
I20260928 22:02:57.175462 20829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:40387 every 8 connection(s)
I20260928 22:02:57.175845 20762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:02:57.176819 20830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:57.179186 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900: Bootstrap starting.
I20260928 22:02:57.179766 20830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:57.180011 20830 log.cc:824] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:57.180926 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900: No bootstrap required, opened a new log
I20260928 22:02:57.181639 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20762
I20260928 22:02:57.181768 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:02:57.181954 2709 external_mini_cluster.cc:1428] Setting key f93d481ea2eabae6d59c0745f524deed
I20260928 22:02:57.183115 20830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553897d47b6d4743b50fe5243609d900" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40387 } }
I20260928 22:02:57.183310 20830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:57.183349 20830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553897d47b6d4743b50fe5243609d900, State: Initialized, Role: FOLLOWER
I20260928 22:02:57.183496 20830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553897d47b6d4743b50fe5243609d900" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40387 } }
I20260928 22:02:57.183584 20830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:02:57.183619 20830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:02:57.183673 20830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:57.184809 20830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553897d47b6d4743b50fe5243609d900" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40387 } }
I20260928 22:02:57.184906 20830 leader_election.cc:302] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 553897d47b6d4743b50fe5243609d900; no voters:
I20260928 22:02:57.185099 20830 leader_election.cc:288] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:02:57.185122 20835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:57.185241 20835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [term 1 LEADER]: Becoming Leader. State: Replica: 553897d47b6d4743b50fe5243609d900, State: Running, Role: LEADER
I20260928 22:02:57.185330 20835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553897d47b6d4743b50fe5243609d900" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40387 } }
I20260928 22:02:57.185405 20830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:02:57.185652 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 553897d47b6d4743b50fe5243609d900. Latest consensus state: current_term: 1 leader_uuid: "553897d47b6d4743b50fe5243609d900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553897d47b6d4743b50fe5243609d900" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40387 } } }
I20260928 22:02:57.185745 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:57.185921 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "553897d47b6d4743b50fe5243609d900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553897d47b6d4743b50fe5243609d900" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40387 } } }
I20260928 22:02:57.185978 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900 [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:57.186101 20840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:02:57.186689 20840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:02:57.188956 20840 catalog_manager.cc:1347] Generated new cluster ID: fe13e296405d4d4e8f5eed386d5bc089
I20260928 22:02:57.189013 20840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:57.194448 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.182529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.2.165.126:40387)
0928 22:02:57.182756 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.182765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.182791 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:02:57.183088 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.183094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.183162 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.183284 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:57.184121 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.185292 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.186402 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.187066 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.187746 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.190433 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:57.190437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:57.190437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:57.190450 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:57.192498 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.192968 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.192971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.192972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.193143 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.193416 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.193416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.193417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.193756 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:57.193921 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.194041 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.194159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":117,"threads_started":1}
I20260928 22:02:57.194723 20840 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:02:57.195403 20840 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:02:57.199682 20840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900: Generated new TSK 0
I20260928 22:02:57.199885 20840 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 22:02:57.231998 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40387
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:57.316293 20856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:57.316521 20856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:57.316547 20856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:57.317811 20856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:57.317864 20856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:57.317893 20856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:02:57.319401 20856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:57.319684 20856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:57.319926 20856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:57.325381 20864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:57.325408 20862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:57.325441 20856 server_base.cc:1034] running on GCE node
W20260928 22:02:57.325408 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:57.325907 20856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:57.326191 20856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:57.327356 20856 hybrid_clock.cc:648] HybridClock initialized: now 1790632977327350 us; error 35 us; skew 500 ppm
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:57.330227 20856 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:02:57.330687 20856 webserver.cc:460] Webserver started at http://127.2.165.65:43017/ using document root <none> and password file <none>
I20260928 22:02:57.330931 20856 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:57.330999 20856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:57.331158 20856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:57.332226 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "5021cead98fb4ab498d6e3fb802680f2"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "5e8815d789dc8d3410f315f00ec41869"
server_key_iv: "03176a085bc76ac3fd9cfd8a8bf7015d"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.332592 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "5021cead98fb4ab498d6e3fb802680f2"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "5e8815d789dc8d3410f315f00ec41869"
server_key_iv: "03176a085bc76ac3fd9cfd8a8bf7015d"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.333984 20856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:57.334748 20871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.334959 20856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:57.335067 20856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "5021cead98fb4ab498d6e3fb802680f2"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "5e8815d789dc8d3410f315f00ec41869"
server_key_iv: "03176a085bc76ac3fd9cfd8a8bf7015d"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.335160 20856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:57.346828 20856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:57.348994 20856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:57.349175 20856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:57.349427 20856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:57.349757 20856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:57.349810 20856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.349853 20856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:57.349882 20856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.356060 20856 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:45453
I20260928 22:02:57.356109 20984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:45453 every 8 connection(s)
I20260928 22:02:57.356487 20856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:02:57.357173 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20856
I20260928 22:02:57.357290 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:02:57.357525 2709 external_mini_cluster.cc:1428] Setting key 74a23ffda3f6a71e3ad93fda24ee3243
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
I20260928 22:02:57.368069 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.357505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:38019 (local address 127.2.165.126:40387)
0928 22:02:57.357565 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.357567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.357857 (+ 290us) server_negotiation.cc:407] Connection header received
0928 22:02:57.358480 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.358481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.358497 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.358574 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:57.359552 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.360067 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.361374 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.361716 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.361817 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.364754 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:57.364756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:57.364756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:57.364765 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:57.366410 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.366861 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.366862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.366863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.366944 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.367245 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.367246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.367246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.367377 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:57.367522 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.367869 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.367987 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:02:57.368168 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.357455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40387 (local address 127.2.165.65:38019)
0928 22:02:57.357728 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:57.357763 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:57.358382 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:57.358635 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:57.358638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:57.359234 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:57.359446 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.359451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.360170 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.360171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.361279 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.361283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.361827 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.361829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.362019 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.362673 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:57.362686 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:57.364617 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:57.366509 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:57.366517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:57.366519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:57.366790 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:57.367019 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:57.367019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:57.367020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:57.367149 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:57.367549 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:57.367556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:57.367705 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 22:02:57.367804 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 22:02:57.367948 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
I20260928 22:02:57.368969 20985 heartbeater.cc:347] Connected to a master server at 127.2.165.126:40387
I20260928 22:02:57.369098 20985 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:57.369331 20985 heartbeater.cc:511] Master 127.2.165.126:40387 requested a full tablet report, sending...
I20260928 22:02:57.369938 20794 ts_manager.cc:194] Registered new tserver with Master: 5021cead98fb4ab498d6e3fb802680f2 (127.2.165.65:45453)
I20260928 22:02:57.370774 20794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:38019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 22:02:57.384215 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40387
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:57.468050 20990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:57.468322 20990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:57.468361 20990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:57.469624 20990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:57.469702 20990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:57.469748 20990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:02:57.471263 20990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:57.471568 20990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:57.471825 20990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:57.477185 20995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:57.477164 20998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:57.477212 20996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:57.477622 20990 server_base.cc:1034] running on GCE node
I20260928 22:02:57.477903 20990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:57.478183 20990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:57.479370 20990 hybrid_clock.cc:648] HybridClock initialized: now 1790632977479354 us; error 35 us; skew 500 ppm
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:57.482391 20990 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:02:57.482957 20990 webserver.cc:460] Webserver started at http://127.2.165.66:40031/ using document root <none> and password file <none>
I20260928 22:02:57.483238 20990 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:57.483312 20990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:57.483476 20990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:57.484589 20990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "0c71576a18934371ba2ca988be54045c"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "fb10ff6bd777fd21f5a849fc8fb12a79"
server_key_iv: "97765e85d75feb6ce0c78772dd920127"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.485069 20990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "0c71576a18934371ba2ca988be54045c"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "fb10ff6bd777fd21f5a849fc8fb12a79"
server_key_iv: "97765e85d75feb6ce0c78772dd920127"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.486770 20990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:02:57.487736 21005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.487990 20990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:57.488123 20990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "0c71576a18934371ba2ca988be54045c"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "fb10ff6bd777fd21f5a849fc8fb12a79"
server_key_iv: "97765e85d75feb6ce0c78772dd920127"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.488210 20990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:57.508931 20990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:57.511768 20990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:57.511961 20990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:57.512243 20990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:57.512594 20990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:57.512651 20990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.512715 20990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:57.512765 20990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.518887 20990 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:42365
I20260928 22:02:57.518931 21118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:42365 every 8 connection(s)
I20260928 22:02:57.519277 20990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:02:57.519670 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 20990
I20260928 22:02:57.519797 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:02:57.520035 2709 external_mini_cluster.cc:1428] Setting key d13ad541fd5dd70bdf8263d6a59b0053
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
I20260928 22:02:57.530071 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.520067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:49989 (local address 127.2.165.126:40387)
0928 22:02:57.520130 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.520132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.520448 (+ 316us) server_negotiation.cc:407] Connection header received
0928 22:02:57.521135 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.521137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.521149 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.521196 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:57.522333 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.522853 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.523912 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.524186 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.524276 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.526831 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:57.526832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:57.526833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:57.526841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:57.528455 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.529016 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.529018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.529018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.529085 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.529407 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.529408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.529409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.529541 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:57.529599 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.529932 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.530006 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:02:57.530198 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.519997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40387 (local address 127.2.165.66:49989)
0928 22:02:57.520295 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:57.520347 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:57.521028 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:57.521272 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:57.521277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:57.522006 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:57.522217 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.522222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.522954 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.522955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.523798 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.523800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.524286 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.524287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.524422 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.525062 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:57.525071 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:57.526724 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:57.528584 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:57.528596 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:57.528601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:57.528932 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:57.529179 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:57.529180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:57.529180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:57.529336 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:57.529607 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:57.529612 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:57.529760 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:02:57.529851 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:02:57.529986 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260928 22:02:57.530726 21119 heartbeater.cc:347] Connected to a master server at 127.2.165.126:40387
I20260928 22:02:57.530841 21119 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:57.531072 21119 heartbeater.cc:511] Master 127.2.165.126:40387 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
I20260928 22:02:57.531479 20794 ts_manager.cc:194] Registered new tserver with Master: 0c71576a18934371ba2ca988be54045c (127.2.165.66:42365)
I20260928 22:02:57.532006 20794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:49989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 22:02:57.548426 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40387
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:02:57.632735 21124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:57.632961 21124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:57.632987 21124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:57.634214 21124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:57.634263 21124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:57.634320 21124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:02:57.635782 21124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:57.636031 21124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:57.636281 21124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:57.645098 21130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:57.645591 21129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:57.645632 21132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:57.645888 21124 server_base.cc:1034] running on GCE node
I20260928 22:02:57.646109 21124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:57.646379 21124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:57.647547 21124 hybrid_clock.cc:648] HybridClock initialized: now 1790632977647531 us; error 30 us; skew 500 ppm
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:02:57.650296 21124 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:02:57.650810 21124 webserver.cc:460] Webserver started at http://127.2.165.67:34895/ using document root <none> and password file <none>
I20260928 22:02:57.651006 21124 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:57.651049 21124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:57.651139 21124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:57.652166 21124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "1273c4f36811418e95315edf19fc5b91"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "b8756066849563f6c2949fa9ae195340"
server_key_iv: "2b3ae57329451bcecb188a4a74d70919"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.652473 21124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "1273c4f36811418e95315edf19fc5b91"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "b8756066849563f6c2949fa9ae195340"
server_key_iv: "2b3ae57329451bcecb188a4a74d70919"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.653879 21124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:02:57.654639 21139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.654886 21124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:57.654994 21124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "1273c4f36811418e95315edf19fc5b91"
format_stamp: "Formatted at 2026-09-28 22:02:57 on dist-test-slave-3t36"
server_key: "b8756066849563f6c2949fa9ae195340"
server_key_iv: "2b3ae57329451bcecb188a4a74d70919"
server_key_version: "encryptionkey@0"
I20260928 22:02:57.655076 21124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:57.674207 21124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:57.676608 21124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:57.676815 21124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:57.677101 21124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:57.677435 21124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:57.677470 21124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.677493 21124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:57.677541 21124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:57.683857 21124 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:44787
I20260928 22:02:57.683900 21252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:44787 every 8 connection(s)
I20260928 22:02:57.684278 21124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:02:57 dist-test-slave-3t36 krb5kdc[20749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:02:57.693868 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 21124
I20260928 22:02:57.694000 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:02:57.694164 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.685127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:32901 (local address 127.2.165.126:40387)
0928 22:02:57.685200 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.685202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.685394 (+ 192us) server_negotiation.cc:407] Connection header received
0928 22:02:57.685958 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.685960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.685973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.686022 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:57.687132 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.687650 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.688685 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.688890 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.688977 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.691183 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:57.691184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:57.691184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:57.691191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:57.692568 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.693211 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.693212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.693212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.693266 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.693482 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.693482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.693483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.693599 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:57.693653 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.693961 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.694062 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:02:57.694211 2709 external_mini_cluster.cc:1428] Setting key 925f4a4caebf49dce8beb5838433796a
I20260928 22:02:57.694383 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.685076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40387 (local address 127.2.165.67:32901)
0928 22:02:57.685260 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:57.685305 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:57.685836 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:57.686114 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:57.686118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:57.686803 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:57.687018 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.687023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.687749 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.687751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.688563 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.688565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.688986 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.688987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.689119 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.689629 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:02:57.689638 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:57.691052 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:57.692647 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:57.692653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:57.692654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:57.693135 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:57.693329 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:02:57.693330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:02:57.693330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:02:57.693428 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:02:57.693671 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:57.693677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:57.693805 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 22:02:57.693908 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 22:02:57.694107 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260928 22:02:57.694978 21253 heartbeater.cc:347] Connected to a master server at 127.2.165.126:40387
I20260928 22:02:57.695117 21253 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:57.695293 21253 heartbeater.cc:511] Master 127.2.165.126:40387 requested a full tablet report, sending...
I20260928 22:02:57.695649 20794 ts_manager.cc:194] Registered new tserver with Master: 1273c4f36811418e95315edf19fc5b91 (127.2.165.67:44787)
I20260928 22:02:57.696036 20794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:32901
I20260928 22:02:57.706188 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:02:57.714221 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.707603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.2.165.126:40387)
0928 22:02:57.707697 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.707700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.707818 (+ 118us) server_negotiation.cc:407] Connection header received
0928 22:02:57.707908 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.707909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.707920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.707978 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:57.708562 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.709128 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.709867 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.710080 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.710172 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.711452 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:57.711454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:57.711454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:02:57.711462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:57.712943 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.713358 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.713360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.713360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.713420 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:02:57.713650 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:02:57.713651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:02:57.713651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:02:57.713797 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:57.713917 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.714038 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.714103 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:02:57.715951 20794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:02:57.717736 20794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:02:57.724696 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.720160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:45453 (local address 127.0.0.1:39900)
0928 22:02:57.720353 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:57.720397 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:57.720465 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:57.721045 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:57.721048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:57.721051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:57.721130 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.721132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.722255 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.722256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.723408 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.723411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.724360 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.724361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.724444 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.724459 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:02:57.724537 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:02:57.724593 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260928 22:02:57.724741 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.720642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.2.165.65:45453)
0928 22:02:57.720846 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.720851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.720862 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:02:57.720910 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.720911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.720924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.721022 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:57.721192 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.722150 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.723500 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.724246 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.724325 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.724387 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.724565 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.724606 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":134,"threads_started":1}
I20260928 22:02:57.725035 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.720168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:44787 (local address 127.0.0.1:39742)
0928 22:02:57.720512 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:57.720523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:57.720581 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:57.721329 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:57.721330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:57.721332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:57.721376 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.721378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.722805 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.722806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.723851 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.723854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.724830 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.724831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.724903 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.724907 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:57.724936 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 22:02:57.724971 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":21,"thread_start_us":129,"threads_started":1}
I20260928 22:02:57.725260 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.720420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42365 (local address 127.0.0.1:56356)
0928 22:02:57.720974 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:57.720983 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:57.721037 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:57.721349 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:57.721350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:57.721351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:57.721378 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.721379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.722798 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.722799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.723952 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:57.723955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.725015 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.725016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:57.725077 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.725081 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:57.725134 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:02:57.725187 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":44,"thread_start_us":74,"threads_started":1}
I20260928 22:02:57.725430 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.720334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.2.165.67:44787)
0928 22:02:57.721164 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.721168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.721178 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:02:57.721215 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.721218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.721233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.721453 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:57.721520 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.722686 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.723955 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.724732 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.725196 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.725263 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.725289 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.725337 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":95,"threads_started":1}
I20260928 22:02:57.725544 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:57.720789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.2.165.66:42365)
0928 22:02:57.720952 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:57.720957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:57.721116 (+ 159us) server_negotiation.cc:407] Connection header received
0928 22:02:57.721174 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:57.721176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:57.721194 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:57.721269 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:57.721470 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.722663 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.724033 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:57.724852 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:57.724915 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:57.724976 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:57.725424 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 22:02:57.725467 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260928 22:02:57.725559 20919 tablet_service.cc:1467] Processing CreateTablet for tablet e5455b1ca9b4443fbccd5050772c300f (DEFAULT_TABLE table=test-table [id=d2a4f6baa7fc46f2a678d5462f4bb7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:57.725972 20919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5455b1ca9b4443fbccd5050772c300f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:57.726254 21053 tablet_service.cc:1467] Processing CreateTablet for tablet e5455b1ca9b4443fbccd5050772c300f (DEFAULT_TABLE table=test-table [id=d2a4f6baa7fc46f2a678d5462f4bb7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:57.726565 21053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5455b1ca9b4443fbccd5050772c300f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:57.728626 21270 tablet_bootstrap.cc:492] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: Bootstrap starting.
I20260928 22:02:57.729220 21271 tablet_bootstrap.cc:492] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Bootstrap starting.
I20260928 22:02:57.729257 21187 tablet_service.cc:1467] Processing CreateTablet for tablet e5455b1ca9b4443fbccd5050772c300f (DEFAULT_TABLE table=test-table [id=d2a4f6baa7fc46f2a678d5462f4bb7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:57.729668 21187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5455b1ca9b4443fbccd5050772c300f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:57.729756 21270 tablet_bootstrap.cc:654] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:57.729897 21271 tablet_bootstrap.cc:654] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:57.730360 21271 log.cc:824] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:57.730404 21270 log.cc:824] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:57.731051 21271 tablet_bootstrap.cc:492] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: No bootstrap required, opened a new log
I20260928 22:02:57.731140 21271 ts_tablet_manager.cc:1397] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:02:57.731166 21270 tablet_bootstrap.cc:492] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: No bootstrap required, opened a new log
I20260928 22:02:57.731238 21270 ts_tablet_manager.cc:1397] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:02:57.732946 21271 raft_consensus.cc:348] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:57.733150 21271 raft_consensus.cc:374] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:57.733199 21271 raft_consensus.cc:729] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c71576a18934371ba2ca988be54045c, State: Initialized, Role: FOLLOWER
I20260928 22:02:57.733310 21271 consensus_queue.cc:260] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:57.733335 21274 tablet_bootstrap.cc:492] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Bootstrap starting.
I20260928 22:02:57.733613 21271 ts_tablet_manager.cc:1428] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:02:57.733530 21270 raft_consensus.cc:348] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:57.733675 21270 raft_consensus.cc:374] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:57.733745 21270 raft_consensus.cc:729] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5021cead98fb4ab498d6e3fb802680f2, State: Initialized, Role: FOLLOWER
I20260928 22:02:57.733762 21119 heartbeater.cc:503] Master 127.2.165.126:40387 was elected leader, sending a full tablet report...
I20260928 22:02:57.733862 21270 consensus_queue.cc:260] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:57.734154 21270 ts_tablet_manager.cc:1428] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:02:57.734181 21274 tablet_bootstrap.cc:654] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:57.734166 20985 heartbeater.cc:503] Master 127.2.165.126:40387 was elected leader, sending a full tablet report...
I20260928 22:02:57.734717 21274 log.cc:824] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:57.735514 21274 tablet_bootstrap.cc:492] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: No bootstrap required, opened a new log
I20260928 22:02:57.735574 21274 ts_tablet_manager.cc:1397] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:02:57.737121 21274 raft_consensus.cc:348] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:57.737250 21274 raft_consensus.cc:374] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:57.737272 21274 raft_consensus.cc:729] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1273c4f36811418e95315edf19fc5b91, State: Initialized, Role: FOLLOWER
I20260928 22:02:57.737401 21274 consensus_queue.cc:260] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:57.737679 21274 ts_tablet_manager.cc:1428] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:02:57.737749 21253 heartbeater.cc:503] Master 127.2.165.126:40387 was elected leader, sending a full tablet report...
W20260928 22:02:57.769853 21120 tablet.cc:2369] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:02:57.857374 20986 tablet.cc:2369] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:02:57.934888 21254 tablet.cc:2369] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:02:58.052073 21276 raft_consensus.cc:482] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:58.052270 21276 raft_consensus.cc:504] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:58.052749 21276 leader_election.cc:288] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1273c4f36811418e95315edf19fc5b91 (127.2.165.67:44787), 0c71576a18934371ba2ca988be54045c (127.2.165.66:42365)
I20260928 22:02:58.057715 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.053602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:58913 (local address 127.2.165.66:42365)
0928 22:02:58.053679 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.053682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.053693 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:02:58.053720 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.053722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.053732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.053794 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:58.053956 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.054768 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.056556 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.057209 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.057312 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.057321 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.057583 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.057634 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 22:02:58.057715 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.053174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:58893 (local address 127.2.165.67:44787)
0928 22:02:58.053257 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.053259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.053520 (+ 261us) server_negotiation.cc:407] Connection header received
0928 22:02:58.053569 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.053570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.053581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.053673 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:58.053906 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.054715 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.056557 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.057186 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.057287 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.057300 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.057595 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.057648 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:02:58.057705 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.053433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:42365 (local address 127.2.165.65:58913)
0928 22:02:58.053610 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.053619 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.053660 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.053810 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.053810 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.053812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:58.053879 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.053880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.054865 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.054866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.056399 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.056404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.057346 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.057347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.057438 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.057441 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.057537 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.057577 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":83,"thread_start_us":63,"threads_started":1}
I20260928 22:02:58.057703 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.053026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:44787 (local address 127.2.165.65:58893)
0928 22:02:58.053313 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.053328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.053486 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.053709 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.053710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.053712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:58.053823 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.053825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.054865 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.054866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.056399 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.056404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.057326 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.057328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.057417 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.057425 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.057508 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.057572 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":85,"thread_start_us":244,"threads_started":1}
I20260928 22:02:58.058288 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5455b1ca9b4443fbccd5050772c300f" candidate_uuid: "5021cead98fb4ab498d6e3fb802680f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1273c4f36811418e95315edf19fc5b91" is_pre_election: true
I20260928 22:02:58.058390 21073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5455b1ca9b4443fbccd5050772c300f" candidate_uuid: "5021cead98fb4ab498d6e3fb802680f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c71576a18934371ba2ca988be54045c" is_pre_election: true
I20260928 22:02:58.058585 21207 raft_consensus.cc:2454] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5021cead98fb4ab498d6e3fb802680f2 in term 0.
I20260928 22:02:58.058598 21073 raft_consensus.cc:2454] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5021cead98fb4ab498d6e3fb802680f2 in term 0.
I20260928 22:02:58.058862 20874 leader_election.cc:302] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c71576a18934371ba2ca988be54045c, 5021cead98fb4ab498d6e3fb802680f2; no voters:
I20260928 22:02:58.059005 21276 raft_consensus.cc:2781] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:02:58.059104 21276 raft_consensus.cc:482] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:02:58.059137 21276 raft_consensus.cc:3037] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:58.059916 21276 raft_consensus.cc:504] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:58.060081 21276 leader_election.cc:288] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [CANDIDATE]: Term 1 election: Requested vote from peers 1273c4f36811418e95315edf19fc5b91 (127.2.165.67:44787), 0c71576a18934371ba2ca988be54045c (127.2.165.66:42365)
I20260928 22:02:58.060402 21073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5455b1ca9b4443fbccd5050772c300f" candidate_uuid: "5021cead98fb4ab498d6e3fb802680f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c71576a18934371ba2ca988be54045c"
I20260928 22:02:58.060402 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5455b1ca9b4443fbccd5050772c300f" candidate_uuid: "5021cead98fb4ab498d6e3fb802680f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1273c4f36811418e95315edf19fc5b91"
I20260928 22:02:58.060518 21073 raft_consensus.cc:3037] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:58.060559 21207 raft_consensus.cc:3037] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:58.061534 21207 raft_consensus.cc:2454] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5021cead98fb4ab498d6e3fb802680f2 in term 1.
I20260928 22:02:58.061548 21073 raft_consensus.cc:2454] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5021cead98fb4ab498d6e3fb802680f2 in term 1.
I20260928 22:02:58.061776 20874 leader_election.cc:302] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c71576a18934371ba2ca988be54045c, 5021cead98fb4ab498d6e3fb802680f2; no voters:
I20260928 22:02:58.061867 21276 raft_consensus.cc:2781] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:58.062090 21276 raft_consensus.cc:686] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5021cead98fb4ab498d6e3fb802680f2, State: Running, Role: LEADER
I20260928 22:02:58.062207 21276 consensus_queue.cc:237] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } }
I20260928 22:02:58.063180 20793 catalog_manager.cc:5495] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5021cead98fb4ab498d6e3fb802680f2 (127.2.165.65). New cstate: current_term: 1 leader_uuid: "5021cead98fb4ab498d6e3fb802680f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5021cead98fb4ab498d6e3fb802680f2" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 45453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 } health_report { overall_health: UNKNOWN } } }
I20260928 22:02:58.079167 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.076825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.2.165.65:45453)
0928 22:02:58.076886 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.076888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.076896 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:02:58.076932 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.076933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.076952 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.077021 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:02:58.077144 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.077613 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.078356 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.078544 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.078610 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.078854 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:02:58.078950 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:02:58.079046 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.079087 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.079103 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:02:58.081408 21207 raft_consensus.cc:1261] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 1 FOLLOWER]: Refusing update from remote peer 5021cead98fb4ab498d6e3fb802680f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:58.081408 21073 raft_consensus.cc:1261] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 1 FOLLOWER]: Refusing update from remote peer 5021cead98fb4ab498d6e3fb802680f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:58.081830 21281 consensus_queue.cc:1035] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c71576a18934371ba2ca988be54045c" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 42365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:58.081977 21276 consensus_queue.cc:1035] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1273c4f36811418e95315edf19fc5b91" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:58.087246 20793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:58.087371 20793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:58.088222 20793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 553897d47b6d4743b50fe5243609d900: Sending DeleteTablet for 3 replicas of tablet e5455b1ca9b4443fbccd5050772c300f
I20260928 22:02:58.088617 20919 tablet_service.cc:1514] Processing DeleteTablet for tablet e5455b1ca9b4443fbccd5050772c300f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:58 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:39900
I20260928 22:02:58.088805 21187 tablet_service.cc:1514] Processing DeleteTablet for tablet e5455b1ca9b4443fbccd5050772c300f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:58 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:39742
I20260928 22:02:58.088907 21294 tablet_replica.cc:330] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2: stopping tablet replica
I20260928 22:02:58.089042 21294 raft_consensus.cc:2229] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:02:58.089140 21053 tablet_service.cc:1514] Processing DeleteTablet for tablet e5455b1ca9b4443fbccd5050772c300f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:58 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:56356
I20260928 22:02:58.089205 21294 raft_consensus.cc:2258] T e5455b1ca9b4443fbccd5050772c300f P 5021cead98fb4ab498d6e3fb802680f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:58.089238 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20856
I20260928 22:02:58.091500 21296 tablet_replica.cc:330] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: stopping tablet replica
I20260928 22:02:58.091631 21296 raft_consensus.cc:2229] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:58.091769 21296 raft_consensus.cc:2258] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:58.092320 21296 ts_tablet_manager.cc:1905] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:58.092854 21295 tablet_replica.cc:330] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: stopping tablet replica
I20260928 22:02:58.093003 21295 raft_consensus.cc:2229] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:58.093164 21295 raft_consensus.cc:2258] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:58.093624 21295 ts_tablet_manager.cc:1905] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:58.094394 21296 ts_tablet_manager.cc:1918] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:58.094476 21296 log.cc:1196] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/e5455b1ca9b4443fbccd5050772c300f
I20260928 22:02:58.094722 21296 ts_tablet_manager.cc:1939] T e5455b1ca9b4443fbccd5050772c300f P 0c71576a18934371ba2ca988be54045c: Deleting consensus metadata
I20260928 22:02:58.095158 21295 ts_tablet_manager.cc:1918] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:58.095248 21295 log.cc:1196] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/e5455b1ca9b4443fbccd5050772c300f
I20260928 22:02:58.095530 20780 catalog_manager.cc:4841] TS 0c71576a18934371ba2ca988be54045c (127.2.165.66:42365): tablet e5455b1ca9b4443fbccd5050772c300f (table test-table [id=d2a4f6baa7fc46f2a678d5462f4bb7f3]) successfully deleted
I20260928 22:02:58.095638 21295 ts_tablet_manager.cc:1939] T e5455b1ca9b4443fbccd5050772c300f P 1273c4f36811418e95315edf19fc5b91: Deleting consensus metadata
W20260928 22:02:58.095806 20781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:45453 (error 108)
I20260928 22:02:58.096063 20780 catalog_manager.cc:4841] TS 1273c4f36811418e95315edf19fc5b91 (127.2.165.67:44787): tablet e5455b1ca9b4443fbccd5050772c300f (table test-table [id=d2a4f6baa7fc46f2a678d5462f4bb7f3]) successfully deleted
I20260928 22:02:58.096380 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.096233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:45453 (local address 127.0.0.1:39928)
0928 22:02:58.096302 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.096348 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:45453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 22:02:58.096455 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20990
W20260928 22:02:58.096524 20781 catalog_manager.cc:4584] TS 5021cead98fb4ab498d6e3fb802680f2 (127.2.165.65:45453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5455b1ca9b4443fbccd5050772c300f: Network error: Client connection negotiation failed: client connection to 127.2.165.65:45453: connect: Connection refused (error 111)
I20260928 22:02:58.101720 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 21124
I20260928 22:02:58.106674 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 20762
2026-09-28T22:02:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2247 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T22:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:02:58Z Disabled control of system clock
I20260928 22:02:58.135663 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46333
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:46333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:02:58.221138 21301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:58.221380 21301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:58.221403 21301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:58.222666 21301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:02:58.222718 21301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:58.222730 21301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:58.222744 21301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:02:58.222756 21301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:02:58.224164 21301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:46333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:58.224466 21301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:58.224841 21301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:58.229910 21307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:58.229928 21301 server_base.cc:1034] running on GCE node
W20260928 22:02:58.229916 21306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:58.229910 21309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:58.230585 21301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:58.230896 21301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:58.232065 21301 hybrid_clock.cc:648] HybridClock initialized: now 1790632978232039 us; error 37 us; skew 500 ppm
I20260928 22:02:58.241662 21301 webserver.cc:460] Webserver started at http://127.2.165.126:46373/ using document root <none> and password file <none>
I20260928 22:02:58.242050 21301 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:58.242108 21301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:58.242266 21301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:58.243475 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "7db51e7d362742aa952e56c46713b13f"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "687e074b256bb79c6ec77863c4a300d3"
server_key_iv: "e5a51291d6ba24c70a54f85e7358a6fb"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.243858 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "7db51e7d362742aa952e56c46713b13f"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "687e074b256bb79c6ec77863c4a300d3"
server_key_iv: "e5a51291d6ba24c70a54f85e7358a6fb"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.245394 21301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:02:58.246343 21315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.246608 21301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:58.246747 21301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "7db51e7d362742aa952e56c46713b13f"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "687e074b256bb79c6ec77863c4a300d3"
server_key_iv: "e5a51291d6ba24c70a54f85e7358a6fb"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.246832 21301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:58.261281 21301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:58.261646 21301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:58.261812 21301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:58.265986 21301 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:46333
I20260928 22:02:58.266026 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:46333 every 8 connection(s)
I20260928 22:02:58.266415 21301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:02:58.267469 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:58.270181 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f: Bootstrap starting.
I20260928 22:02:58.270797 21368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:58.271044 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 21301
I20260928 22:02:58.271158 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:02:58.271165 21368 log.cc:824] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:58.271365 2709 external_mini_cluster.cc:1428] Setting key 42542d610f419db644ed5249ee892af9
I20260928 22:02:58.272009 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f: No bootstrap required, opened a new log
I20260928 22:02:58.274974 21368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db51e7d362742aa952e56c46713b13f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46333 } }
I20260928 22:02:58.275177 21368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:58.275219 21368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db51e7d362742aa952e56c46713b13f, State: Initialized, Role: FOLLOWER
I20260928 22:02:58.275379 21368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db51e7d362742aa952e56c46713b13f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46333 } }
I20260928 22:02:58.275471 21368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:02:58.275503 21368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:02:58.275544 21368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:58.276364 21368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db51e7d362742aa952e56c46713b13f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46333 } }
I20260928 22:02:58.276491 21368 leader_election.cc:302] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db51e7d362742aa952e56c46713b13f; no voters:
I20260928 22:02:58.276774 21368 leader_election.cc:288] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:02:58.276804 21373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:58.276930 21373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [term 1 LEADER]: Becoming Leader. State: Replica: 7db51e7d362742aa952e56c46713b13f, State: Running, Role: LEADER
I20260928 22:02:58.277264 21373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db51e7d362742aa952e56c46713b13f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46333 } }
I20260928 22:02:58.277319 21368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:02:58.277702 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.271902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.2.165.126:46333)
0928 22:02:58.272138 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.272145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.272174 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:02:58.272643 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.272651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.272776 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.272895 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:58.273147 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.274026 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.275324 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.276007 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.276174 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.276928 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:58.276934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:58.276934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:58.277158 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:58.277204 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:58.277220 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:58.277256 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.277381 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.277468 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
I20260928 22:02:58.277793 21374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7db51e7d362742aa952e56c46713b13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db51e7d362742aa952e56c46713b13f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46333 } } }
I20260928 22:02:58.277884 21374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:58.278208 21375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db51e7d362742aa952e56c46713b13f. Latest consensus state: current_term: 1 leader_uuid: "7db51e7d362742aa952e56c46713b13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db51e7d362742aa952e56c46713b13f" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 46333 } } }
I20260928 22:02:58.278287 21375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f [sys.catalog]: This master's current role is: LEADER
I20260928 22:02:58.278412 21379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:02:58.278947 21379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:02:58.280845 21379 catalog_manager.cc:1347] Generated new cluster ID: f1b2ae112b7249d69cfb8366290bddbf
I20260928 22:02:58.280934 21379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:02:58.304958 21379 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:02:58.305603 21379 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:02:58.314406 21379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f: Generated new TSK 0
I20260928 22:02:58.314680 21379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:02:58.323951 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46333
--builtin_ntp_servers=127.2.165.84:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:02:58.412058 21392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:58.412295 21392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:58.412319 21392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:58.413605 21392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:58.413663 21392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:58.413692 21392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:02:58.415195 21392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:58.415555 21392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:58.416003 21392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:58.421700 21400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:58.421700 21397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:58.421851 21398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:58.422235 21392 server_base.cc:1034] running on GCE node
I20260928 22:02:58.422480 21392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:58.422775 21392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:58.424003 21392 hybrid_clock.cc:648] HybridClock initialized: now 1790632978423986 us; error 34 us; skew 500 ppm
I20260928 22:02:58.425549 21392 webserver.cc:460] Webserver started at http://127.2.165.65:38895/ using document root <none> and password file <none>
I20260928 22:02:58.425834 21392 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:58.425910 21392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:58.426103 21392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:58.427342 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "0b7bd7980b904a40b7da92c5d9db67b6"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "8337cdfabc65f2e24fbad9fb13d4cfc9"
server_key_iv: "54a784fdb77b51ef2cbfd2a89d107f12"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.427664 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "0b7bd7980b904a40b7da92c5d9db67b6"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "8337cdfabc65f2e24fbad9fb13d4cfc9"
server_key_iv: "54a784fdb77b51ef2cbfd2a89d107f12"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.429131 21392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:02:58.430083 21406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.430341 21392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:58.430445 21392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "0b7bd7980b904a40b7da92c5d9db67b6"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "8337cdfabc65f2e24fbad9fb13d4cfc9"
server_key_iv: "54a784fdb77b51ef2cbfd2a89d107f12"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.430523 21392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:58.471297 21392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:58.471670 21392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:58.471839 21392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:58.472114 21392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:58.472471 21392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:58.472530 21392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.472574 21392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:58.472604 21392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.478842 21392 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:40319
I20260928 22:02:58.478953 21519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:40319 every 8 connection(s)
I20260928 22:02:58.479301 21392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:02:58.480278 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 21392
I20260928 22:02:58.480396 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:02:58.480806 2709 external_mini_cluster.cc:1428] Setting key a91de7d0964fd8c86590f3d139fee5e3
I20260928 22:02:58.481964 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46333
--builtin_ntp_servers=127.2.165.84:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:02:58.485478 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.480118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:47079 (local address 127.2.165.126:46333)
0928 22:02:58.480171 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.480173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.480708 (+ 535us) server_negotiation.cc:407] Connection header received
0928 22:02:58.481314 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.481315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.481326 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.481367 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:58.482153 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.482677 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.483703 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.483927 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.484013 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.485133 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:58.485135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:58.485135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:58.485140 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:58.485157 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:58.485160 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:58.485216 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.485377 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.485410 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:02:58.485744 21522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.480132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46333 (local address 127.2.165.65:47079)
0928 22:02:58.480463 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.480519 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.481204 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.481444 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.481448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.481467 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:58.482005 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.482033 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.482775 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.482775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.483605 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.483608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.484025 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.484026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.484195 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.484931 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:02:58.484932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:58.484975 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:02:58.485007 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:58.485235 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:58.485241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:58.485244 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.485316 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.485444 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260928 22:02:58.486359 21520 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46333
I20260928 22:02:58.486486 21520 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:58.486721 21520 heartbeater.cc:511] Master 127.2.165.126:46333 requested a full tablet report, sending...
I20260928 22:02:58.487403 21332 ts_manager.cc:194] Registered new tserver with Master: 0b7bd7980b904a40b7da92c5d9db67b6 (127.2.165.65:40319)
I20260928 22:02:58.488124 21332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.65:47079
W20260928 22:02:58.570667 21523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:58.570933 21523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:58.570971 21523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:58.572176 21523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:58.572252 21523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:58.572295 21523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:02:58.573730 21523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:58.574046 21523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:58.574373 21523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:58.580375 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:58.580370 21529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:58.580739 21523 server_base.cc:1034] running on GCE node
W20260928 22:02:58.580375 21528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:58.581418 21523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:58.581799 21523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:58.582985 21523 hybrid_clock.cc:648] HybridClock initialized: now 1790632978582967 us; error 36 us; skew 500 ppm
I20260928 22:02:58.584632 21523 webserver.cc:460] Webserver started at http://127.2.165.66:41097/ using document root <none> and password file <none>
I20260928 22:02:58.584946 21523 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:58.585000 21523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:58.585143 21523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:58.586333 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "cd8395e7c8774d98aeba5221456c0b0a"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "bad8ad9eb62144188fe9f1bc47278a68"
server_key_iv: "6ad669763770c02895cc535bccdb23b7"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.586705 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "cd8395e7c8774d98aeba5221456c0b0a"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "bad8ad9eb62144188fe9f1bc47278a68"
server_key_iv: "6ad669763770c02895cc535bccdb23b7"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.588156 21523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:02:58.589123 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.589371 21523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:02:58.589491 21523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "cd8395e7c8774d98aeba5221456c0b0a"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "bad8ad9eb62144188fe9f1bc47278a68"
server_key_iv: "6ad669763770c02895cc535bccdb23b7"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.589581 21523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:58.601729 21523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:58.602077 21523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:58.602264 21523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:58.602541 21523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:58.602900 21523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:58.602958 21523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.603003 21523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:58.603046 21523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.609128 21523 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:46709
I20260928 22:02:58.609207 21650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:46709 every 8 connection(s)
I20260928 22:02:58.609577 21523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:02:58.615319 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.610460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:52123 (local address 127.2.165.126:46333)
0928 22:02:58.610552 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.610555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.610759 (+ 204us) server_negotiation.cc:407] Connection header received
0928 22:02:58.611319 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.611320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.611333 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.611388 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:58.612133 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.612645 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.613735 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.613956 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.614040 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.615010 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:58.615012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:58.615013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:58.615018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:58.615034 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:58.615038 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:58.615087 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.615204 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.615234 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 22:02:58.615576 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.610361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46333 (local address 127.2.165.66:52123)
0928 22:02:58.610622 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.610669 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.611208 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.611476 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.611480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.611493 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:58.612017 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.612021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.612794 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.612795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.613637 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.613640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.614049 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.614050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.614203 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.614861 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:02:58.614861 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:58.614891 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:02:58.614915 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:58.615087 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:58.615092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:58.615121 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.615208 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.615315 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260928 22:02:58.616196 21651 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46333
I20260928 22:02:58.616283 21651 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:58.616487 21651 heartbeater.cc:511] Master 127.2.165.126:46333 requested a full tablet report, sending...
I20260928 22:02:58.617060 21332 ts_manager.cc:194] Registered new tserver with Master: cd8395e7c8774d98aeba5221456c0b0a (127.2.165.66:46709)
I20260928 22:02:58.617489 21332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.66:52123
I20260928 22:02:58.617987 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 21523
I20260928 22:02:58.618116 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:02:58.618326 2709 external_mini_cluster.cc:1428] Setting key 90f287b49c0b6e32a5c3db966d0da042
I20260928 22:02:58.619273 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46333
--builtin_ntp_servers=127.2.165.84:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:02:58.701774 21654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:02:58.701993 21654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:02:58.702015 21654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:02:58.703128 21654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:02:58.703176 21654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:02:58.703200 21654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:02:58.704480 21654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:46333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:02:58.704749 21654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:02:58.705034 21654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:02:58.710242 21660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:58.710453 21662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:02:58.710482 21659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:02:58.710631 21654 server_base.cc:1034] running on GCE node
I20260928 22:02:58.710850 21654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:02:58.711117 21654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:02:58.712308 21654 hybrid_clock.cc:648] HybridClock initialized: now 1790632978712299 us; error 45 us; skew 500 ppm
I20260928 22:02:58.713826 21654 webserver.cc:460] Webserver started at http://127.2.165.67:35725/ using document root <none> and password file <none>
I20260928 22:02:58.714111 21654 fs_manager.cc:359] Metadata directory not provided
I20260928 22:02:58.714188 21654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:02:58.714334 21654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:02:58.715461 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "319b0b74026e4fe5998fce8ce54c6a21"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "58f4f610dd45ddae08585b1ec748bb58"
server_key_iv: "e65cb1081aa2778acd67551f5295d9a6"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.715842 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "319b0b74026e4fe5998fce8ce54c6a21"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "58f4f610dd45ddae08585b1ec748bb58"
server_key_iv: "e65cb1081aa2778acd67551f5295d9a6"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.717350 21654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:02:58.718199 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.718403 21654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:02:58.718509 21654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "319b0b74026e4fe5998fce8ce54c6a21"
format_stamp: "Formatted at 2026-09-28 22:02:58 on dist-test-slave-3t36"
server_key: "58f4f610dd45ddae08585b1ec748bb58"
server_key_iv: "e65cb1081aa2778acd67551f5295d9a6"
server_key_version: "encryptionkey@0"
I20260928 22:02:58.718593 21654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:02:58.732100 21654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:02:58.732434 21654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:02:58.732583 21654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:02:58.732893 21654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:02:58.733316 21654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:02:58.733381 21654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.733419 21654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:02:58.733446 21654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:02:58.739903 21654 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:36819
I20260928 22:02:58.739948 21781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:36819 every 8 connection(s)
I20260928 22:02:58.740295 21654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:02:58.744295 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 21654
I20260928 22:02:58.744416 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:02:58.744644 2709 external_mini_cluster.cc:1428] Setting key 72dedc3af76ff78422727134ed629172
I20260928 22:02:58.746294 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.741087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:50363 (local address 127.2.165.126:46333)
0928 22:02:58.741190 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.741193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.741390 (+ 197us) server_negotiation.cc:407] Connection header received
0928 22:02:58.741966 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.741968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.741981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.742041 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:58.742817 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.743348 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.744558 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.744864 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.744930 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.746032 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:58.746033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:58.746033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:58.746037 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:58.746051 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:58.746054 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:58.746117 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.746170 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.746194 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 22:02:58.746482 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.741068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:46333 (local address 127.2.165.67:50363)
0928 22:02:58.741263 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.741299 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.741853 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.742117 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.742120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.742128 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:02:58.742710 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.742715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.743450 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.743451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.744463 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.744467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.744961 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.744963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.745128 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.745869 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:02:58.745870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:02:58.745901 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:02:58.745949 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:02:58.746113 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:02:58.746118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:02:58.746120 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.746174 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.746258 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260928 22:02:58.747025 21782 heartbeater.cc:347] Connected to a master server at 127.2.165.126:46333
I20260928 22:02:58.747131 21782 heartbeater.cc:464] Registering TS with master...
I20260928 22:02:58.747318 21782 heartbeater.cc:511] Master 127.2.165.126:46333 requested a full tablet report, sending...
I20260928 22:02:58.747732 21332 ts_manager.cc:194] Registered new tserver with Master: 319b0b74026e4fe5998fce8ce54c6a21 (127.2.165.67:36819)
I20260928 22:02:58.748067 21332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.67:50363
I20260928 22:02:58.756912 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:02:58.761580 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.758347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.2.165.126:46333)
0928 22:02:58.758464 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.758467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.758616 (+ 149us) server_negotiation.cc:407] Connection header received
0928 22:02:58.758685 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.758686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.758697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.758746 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:02:58.758916 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.759432 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.760207 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.760414 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.760495 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.761268 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:02:58.761269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:02:58.761270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:02:58.761275 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:02:58.761296 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:02:58.761300 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:02:58.761402 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.761477 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.761489 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 22:02:58.763689 21332 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:02:58.765395 21332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:02:58.772179 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.768000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.2.165.67:36819)
0928 22:02:58.768620 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.768625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.768635 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:02:58.768694 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.768697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.768714 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.768788 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:58.768997 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.769692 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.771270 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.771768 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.771860 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.771973 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.772047 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.772086 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":89,"threads_started":1}
I20260928 22:02:58.772178 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.767836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:40319 (local address 127.0.0.1:48458)
0928 22:02:58.768081 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.768090 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.768149 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.768656 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.768657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.768659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:58.768724 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.768726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.769968 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.769969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.770962 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.770965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.771869 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.771870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.771947 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.771952 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.772000 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.772054 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260928 22:02:58.772341 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.767825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:36819 (local address 127.0.0.1:45220)
0928 22:02:58.768035 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.768227 (+ 192us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.768259 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.768861 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.768861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.768863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:58.768899 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.768900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.769782 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.769783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.771157 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.771160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.771858 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.771860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.771935 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.771949 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.772030 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.772075 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":84,"thread_start_us":76,"threads_started":1}
I20260928 22:02:58.772408 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.768269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.2.165.65:40319)
0928 22:02:58.768483 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.768487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.768499 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:02:58.768538 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.768540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.768556 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.768795 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:58.768855 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.769907 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.771052 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.771770 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.772146 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.772246 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.772273 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.772310 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20260928 22:02:58.772516 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.767825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:46709 (local address 127.0.0.1:54762)
0928 22:02:58.768017 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:58.768061 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:58.768149 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:58.768437 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:58.768441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:58.768443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:58.768527 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.768529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.769907 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.769908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.771157 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:58.771160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.772282 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.772283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:58.772341 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.772344 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:02:58.772393 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:02:58.772439 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":44,"thread_start_us":77,"threads_started":1}
I20260928 22:02:58.773263 21454 tablet_service.cc:1467] Processing CreateTablet for tablet 601b42787a3e4b74b4d1931a5c853a1c (DEFAULT_TABLE table=test-table [id=e08b80992e5344fdbd9cb752f9ad7014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:58.773433 21716 tablet_service.cc:1467] Processing CreateTablet for tablet 601b42787a3e4b74b4d1931a5c853a1c (DEFAULT_TABLE table=test-table [id=e08b80992e5344fdbd9cb752f9ad7014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:58.773671 21454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601b42787a3e4b74b4d1931a5c853a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:58.773816 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601b42787a3e4b74b4d1931a5c853a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:58.776574 21796 tablet_bootstrap.cc:492] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Bootstrap starting.
I20260928 22:02:58.776629 21797 tablet_bootstrap.cc:492] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: Bootstrap starting.
I20260928 22:02:58.776975 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:58.768098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.2.165.66:46709)
0928 22:02:58.768244 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:58.768249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:58.768262 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:02:58.768304 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:58.768307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:58.768326 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:58.768421 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:58.768638 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.769824 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.771266 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:58.772112 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:58.772172 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:58.772246 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:58.772748 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 22:02:58.776873 (+ 4125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 22:02:58.777392 21796 tablet_bootstrap.cc:654] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:58.777436 21797 tablet_bootstrap.cc:654] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:58.777747 21796 log.cc:824] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:58.777701 21585 tablet_service.cc:1467] Processing CreateTablet for tablet 601b42787a3e4b74b4d1931a5c853a1c (DEFAULT_TABLE table=test-table [id=e08b80992e5344fdbd9cb752f9ad7014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:02:58.777820 21797 log.cc:824] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:58.778096 21585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601b42787a3e4b74b4d1931a5c853a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:02:58.778645 21797 tablet_bootstrap.cc:492] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: No bootstrap required, opened a new log
I20260928 22:02:58.778806 21797 ts_tablet_manager.cc:1397] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:02:58.778894 21796 tablet_bootstrap.cc:492] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: No bootstrap required, opened a new log
I20260928 22:02:58.778978 21796 ts_tablet_manager.cc:1397] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:02:58.781191 21800 tablet_bootstrap.cc:492] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: Bootstrap starting.
I20260928 22:02:58.781572 21797 raft_consensus.cc:348] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:58.781762 21797 raft_consensus.cc:374] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:58.781806 21797 raft_consensus.cc:729] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7bd7980b904a40b7da92c5d9db67b6, State: Initialized, Role: FOLLOWER
I20260928 22:02:58.781857 21800 tablet_bootstrap.cc:654] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: Neither blocks nor log segments found. Creating new log.
I20260928 22:02:58.781793 21796 raft_consensus.cc:348] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:58.781957 21796 raft_consensus.cc:374] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:58.782029 21796 raft_consensus.cc:729] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319b0b74026e4fe5998fce8ce54c6a21, State: Initialized, Role: FOLLOWER
I20260928 22:02:58.781991 21797 consensus_queue.cc:260] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:58.782167 21800 log.cc:824] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:02:58.782181 21796 consensus_queue.cc:260] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:58.782261 21797 ts_tablet_manager.cc:1428] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 22:02:58.782401 21520 heartbeater.cc:503] Master 127.2.165.126:46333 was elected leader, sending a full tablet report...
I20260928 22:02:58.782500 21796 ts_tablet_manager.cc:1428] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:02:58.782639 21782 heartbeater.cc:503] Master 127.2.165.126:46333 was elected leader, sending a full tablet report...
I20260928 22:02:58.782945 21800 tablet_bootstrap.cc:492] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: No bootstrap required, opened a new log
I20260928 22:02:58.783023 21800 ts_tablet_manager.cc:1397] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:02:58.785123 21800 raft_consensus.cc:348] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:58.785280 21800 raft_consensus.cc:374] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:02:58.785327 21800 raft_consensus.cc:729] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8395e7c8774d98aeba5221456c0b0a, State: Initialized, Role: FOLLOWER
I20260928 22:02:58.785442 21800 consensus_queue.cc:260] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:58.785691 21800 ts_tablet_manager.cc:1428] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:02:58.785776 21651 heartbeater.cc:503] Master 127.2.165.126:46333 was elected leader, sending a full tablet report...
W20260928 22:02:58.860280 21652 tablet.cc:2369] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:02:58.980875 21521 tablet.cc:2369] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:02:58.990896 21783 tablet.cc:2369] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:02:59.032039 21801 raft_consensus.cc:482] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:02:59.032253 21801 raft_consensus.cc:504] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:59.032694 21801 leader_election.cc:288] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319b0b74026e4fe5998fce8ce54c6a21 (127.2.165.67:36819), cd8395e7c8774d98aeba5221456c0b0a (127.2.165.66:46709)
I20260928 22:02:59.035985 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:59.033076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:46603 (local address 127.2.165.67:36819)
0928 22:02:59.033121 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:59.033124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:59.033165 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:02:59.033294 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:59.033295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:59.033306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:59.033362 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:59.033551 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.034060 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.035185 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.035602 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.035658 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:59.035665 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:59.035853 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 22:02:59.035893 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:02:59.036024 21806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:59.032934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:36819 (local address 127.2.165.65:46603)
0928 22:02:59.033085 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:59.033099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:59.033181 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:59.033373 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:59.033374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:59.033376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:59.033466 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:59.033468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.034175 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.034176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:59.035074 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:59.035078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.035686 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.035687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:59.035762 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:59.035767 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:02:59.035830 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:02:59.035875 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260928 22:02:59.036128 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:59.033022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:54187 (local address 127.2.165.66:46709)
0928 22:02:59.033137 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:59.033139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:59.033219 (+ 80us) server_negotiation.cc:407] Connection header received
0928 22:02:59.033347 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:59.033348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:59.033358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:59.033413 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:02:59.033684 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.034275 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.035299 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.035764 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.035840 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:59.035847 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:59.036003 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 22:02:59.036052 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 22:02:59.036360 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:59.032934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:46709 (local address 127.2.165.65:54187)
0928 22:02:59.033118 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:59.033130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:02:59.033222 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:02:59.033427 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:02:59.033427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:02:59.033429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:02:59.033526 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:59.033528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.034356 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.034388 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:59.035208 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:02:59.035211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.035856 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.035858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:02:59.035922 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:59.035926 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:02:59.035983 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:02:59.036026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":17,"thread_start_us":61,"threads_started":1}
I20260928 22:02:59.036448 21736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601b42787a3e4b74b4d1931a5c853a1c" candidate_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319b0b74026e4fe5998fce8ce54c6a21" is_pre_election: true
I20260928 22:02:59.036659 21736 raft_consensus.cc:2454] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7bd7980b904a40b7da92c5d9db67b6 in term 0.
I20260928 22:02:59.036968 21409 leader_election.cc:302] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7bd7980b904a40b7da92c5d9db67b6, 319b0b74026e4fe5998fce8ce54c6a21; no voters:
I20260928 22:02:59.037004 21605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601b42787a3e4b74b4d1931a5c853a1c" candidate_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8395e7c8774d98aeba5221456c0b0a" is_pre_election: true
I20260928 22:02:59.037137 21801 raft_consensus.cc:2781] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:02:59.037156 21605 raft_consensus.cc:2454] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7bd7980b904a40b7da92c5d9db67b6 in term 0.
I20260928 22:02:59.037205 21801 raft_consensus.cc:482] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:02:59.037241 21801 raft_consensus.cc:3037] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:59.037990 21801 raft_consensus.cc:504] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:59.038141 21801 leader_election.cc:288] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [CANDIDATE]: Term 1 election: Requested vote from peers 319b0b74026e4fe5998fce8ce54c6a21 (127.2.165.67:36819), cd8395e7c8774d98aeba5221456c0b0a (127.2.165.66:46709)
I20260928 22:02:59.038281 21736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601b42787a3e4b74b4d1931a5c853a1c" candidate_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319b0b74026e4fe5998fce8ce54c6a21"
I20260928 22:02:59.038298 21605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601b42787a3e4b74b4d1931a5c853a1c" candidate_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8395e7c8774d98aeba5221456c0b0a"
I20260928 22:02:59.038367 21736 raft_consensus.cc:3037] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:59.038401 21605 raft_consensus.cc:3037] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:02:59.039147 21605 raft_consensus.cc:2454] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7bd7980b904a40b7da92c5d9db67b6 in term 1.
I20260928 22:02:59.039152 21736 raft_consensus.cc:2454] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7bd7980b904a40b7da92c5d9db67b6 in term 1.
I20260928 22:02:59.039367 21410 leader_election.cc:302] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7bd7980b904a40b7da92c5d9db67b6, cd8395e7c8774d98aeba5221456c0b0a; no voters:
I20260928 22:02:59.039479 21801 raft_consensus.cc:2781] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:02:59.039646 21801 raft_consensus.cc:686] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0b7bd7980b904a40b7da92c5d9db67b6, State: Running, Role: LEADER
I20260928 22:02:59.039765 21801 consensus_queue.cc:237] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } }
I20260928 22:02:59.040752 21330 catalog_manager.cc:5495] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7bd7980b904a40b7da92c5d9db67b6 (127.2.165.65). New cstate: current_term: 1 leader_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b7bd7980b904a40b7da92c5d9db67b6" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 40319 } health_report { overall_health: HEALTHY } } }
I20260928 22:02:59.056527 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:59.053721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.2.165.65:40319)
0928 22:02:59.053793 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:02:59.053794 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:02:59.053802 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:02:59.053838 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:02:59.053839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:02:59.053859 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:02:59.053938 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:02:59.054106 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.054581 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.055653 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:02:59.055857 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:02:59.055975 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:02:59.056149 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:02:59.056251 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:02:59.056361 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 22:02:59.056415 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 22:02:59.056428 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:02:59.058768 21736 raft_consensus.cc:1261] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 1 FOLLOWER]: Refusing update from remote peer 0b7bd7980b904a40b7da92c5d9db67b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:59.058774 21605 raft_consensus.cc:1261] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 1 FOLLOWER]: Refusing update from remote peer 0b7bd7980b904a40b7da92c5d9db67b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:02:59.059180 21801 consensus_queue.cc:1035] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8395e7c8774d98aeba5221456c0b0a" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 46709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:59.059351 21807 consensus_queue.cc:1035] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319b0b74026e4fe5998fce8ce54c6a21" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:02:59.062696 21810 mvcc.cc:205] Tried to move back new op lower bound from 7334432682221715456 to 7334432682147753984. Current Snapshot: MvccSnapshot[applied={T|T < 7334432682221715456}]
I20260928 22:02:59.064951 21330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:59.065073 21330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:02:59.065939 21330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7db51e7d362742aa952e56c46713b13f: Sending DeleteTablet for 3 replicas of tablet 601b42787a3e4b74b4d1931a5c853a1c
I20260928 22:02:59.066320 21454 tablet_service.cc:1514] Processing DeleteTablet for tablet 601b42787a3e4b74b4d1931a5c853a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:59 UTC) from {username='slave'} at 127.0.0.1:48458
I20260928 22:02:59.066380 21585 tablet_service.cc:1514] Processing DeleteTablet for tablet 601b42787a3e4b74b4d1931a5c853a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:59 UTC) from {username='slave'} at 127.0.0.1:54762
I20260928 22:02:59.066365 21716 tablet_service.cc:1514] Processing DeleteTablet for tablet 601b42787a3e4b74b4d1931a5c853a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:02:59 UTC) from {username='slave'} at 127.0.0.1:45220
I20260928 22:02:59.066610 21821 tablet_replica.cc:330] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a: stopping tablet replica
I20260928 22:02:59.066725 21820 tablet_replica.cc:330] T 601b42787a3e4b74b4d1931a5c853a1c P 0b7bd7980b904a40b7da92c5d9db67b6: stopping tablet replica
I20260928 22:02:59.066797 21821 raft_consensus.cc:2229] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:59.066870 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 21392
I20260928 22:02:59.066926 21821 raft_consensus.cc:2258] T 601b42787a3e4b74b4d1931a5c853a1c P cd8395e7c8774d98aeba5221456c0b0a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 22:02:59.072407 21316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:40319 (error 108)
I20260928 22:02:59.072558 21822 tablet_replica.cc:330] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: stopping tablet replica
I20260928 22:02:59.072624 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 21523
I20260928 22:02:59.073323 21822 raft_consensus.cc:2229] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:02:59.073465 21822 raft_consensus.cc:2258] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:02:59.074313 21822 ts_tablet_manager.cc:1905] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:02:59.074810 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:59.074625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:40319 (local address 127.0.0.1:48488)
0928 22:02:59.074711 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:59.074770 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 22:02:59.074950 21316 catalog_manager.cc:4584] TS 0b7bd7980b904a40b7da92c5d9db67b6 (127.2.165.65:40319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 601b42787a3e4b74b4d1931a5c853a1c: Network error: Client connection negotiation failed: client connection to 127.2.165.65:40319: connect: Connection refused (error 111)
I20260928 22:02:59.076193 21822 ts_tablet_manager.cc:1918] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:02:59.076259 21822 log.cc:1196] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/601b42787a3e4b74b4d1931a5c853a1c
I20260928 22:02:59.076516 21822 ts_tablet_manager.cc:1939] T 601b42787a3e4b74b4d1931a5c853a1c P 319b0b74026e4fe5998fce8ce54c6a21: Deleting consensus metadata
I20260928 22:02:59.077070 21318 catalog_manager.cc:4841] TS 319b0b74026e4fe5998fce8ce54c6a21 (127.2.165.67:36819): tablet 601b42787a3e4b74b4d1931a5c853a1c (table test-table [id=e08b80992e5344fdbd9cb752f9ad7014]) successfully deleted
I20260928 22:02:59.077893 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 21654
I20260928 22:02:59.077915 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:02:59.077814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:46709 (local address 127.0.0.1:54764)
0928 22:02:59.077853 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:02:59.077880 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.66:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 22:02:59.082752 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 21301
2026-09-28T22:02:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:02:59 dist-test-slave-3t36 krb5kdc[21826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:02:59 dist-test-slave-3t36 krb5kdc[21826](info): set up 2 sockets
Sep 28 22:02:59 dist-test-slave-3t36 krb5kdc[21826](info): commencing operation
krb5kdc: starting...
W20260928 22:03:00.132643 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:00.259639 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:37285
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:37285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:00.342221 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:00.342482 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:00.342509 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:00.343739 21839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:00.343787 21839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:00.343801 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:00.343813 21839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:00.343825 21839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:00.345392 21839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:37285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:00.345678 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:00.345932 21839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:00.350958 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:00.350983 21844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:00.350968 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:00.351240 21839 server_base.cc:1034] running on GCE node
I20260928 22:03:00.351477 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:00.351789 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:00.352993 21839 hybrid_clock.cc:648] HybridClock initialized: now 1790632980352977 us; error 37 us; skew 500 ppm
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:00.355649 21839 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:00.356050 21839 webserver.cc:460] Webserver started at http://127.2.165.126:33873/ using document root <none> and password file <none>
I20260928 22:03:00.356233 21839 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:00.356274 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:00.356364 21839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:00.357486 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "285b19d68e4446a098ccbe6b81195497"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "2bfce35b22dacc8f0784ef58562a7ac7"
server_key_iv: "25dc9199fa21127c535774ba642456ea"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.357802 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "285b19d68e4446a098ccbe6b81195497"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "2bfce35b22dacc8f0784ef58562a7ac7"
server_key_iv: "25dc9199fa21127c535774ba642456ea"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.359153 21839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:03:00.360044 21854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.360262 21839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:00.360353 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "285b19d68e4446a098ccbe6b81195497"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "2bfce35b22dacc8f0784ef58562a7ac7"
server_key_iv: "25dc9199fa21127c535774ba642456ea"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.360412 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:00.377465 21839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:00.379869 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:00.379988 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:00.383962 21839 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:37285
I20260928 22:03:00.384042 21906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:37285 every 8 connection(s)
I20260928 22:03:00.384397 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:00.385000 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 21839
I20260928 22:03:00.385100 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:00.385322 2709 external_mini_cluster.cc:1428] Setting key 01d6c97108f0e6a52daec5727c0050ed
I20260928 22:03:00.385555 21907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:00.389146 21907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497: Bootstrap starting.
I20260928 22:03:00.389788 21907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:00.390136 21907 log.cc:824] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:00.390920 21907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497: No bootstrap required, opened a new log
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:00.393217 21907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b19d68e4446a098ccbe6b81195497" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37285 } }
I20260928 22:03:00.393358 21907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:00.393380 21907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285b19d68e4446a098ccbe6b81195497, State: Initialized, Role: FOLLOWER
I20260928 22:03:00.393553 21907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b19d68e4446a098ccbe6b81195497" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37285 } }
I20260928 22:03:00.393646 21907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:00.393704 21907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:00.393762 21907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:00.394485 21907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b19d68e4446a098ccbe6b81195497" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37285 } }
I20260928 22:03:00.394619 21907 leader_election.cc:302] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 285b19d68e4446a098ccbe6b81195497; no voters:
I20260928 22:03:00.394833 21907 leader_election.cc:288] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:00.394934 21912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:00.395164 21907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:00.396350 21912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [term 1 LEADER]: Becoming Leader. State: Replica: 285b19d68e4446a098ccbe6b81195497, State: Running, Role: LEADER
I20260928 22:03:00.396502 21912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b19d68e4446a098ccbe6b81195497" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37285 } }
I20260928 22:03:00.397080 21924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 285b19d68e4446a098ccbe6b81195497. Latest consensus state: current_term: 1 leader_uuid: "285b19d68e4446a098ccbe6b81195497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b19d68e4446a098ccbe6b81195497" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37285 } } }
I20260928 22:03:00.397226 21913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "285b19d68e4446a098ccbe6b81195497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b19d68e4446a098ccbe6b81195497" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37285 } } }
I20260928 22:03:00.397931 21924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:00.397935 21913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497 [sys.catalog]: This master's current role is: LEADER
W20260928 22:03:00.397889 21926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:00.398133 21926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:00.398222 21929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:00.398439 21929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:00.398490 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.385931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.2.165.126:37285)
0928 22:03:00.386153 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.386162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.386198 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:03:00.386777 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.386787 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.386853 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.387015 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:00.387644 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.389032 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.389878 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.390481 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.390619 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.393790 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:00.393795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:00.393796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:00.393811 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:00.396156 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.396646 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.396650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.396650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.396793 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.397127 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.397128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.397129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.397508 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:00.397682 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.398050 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.398219 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260928 22:03:00.404628 21929 catalog_manager.cc:1347] Generated new cluster ID: 145289d5d950408ab0ca645ab1104bd2
I20260928 22:03:00.404759 21929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:00.411542 21929 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:00.412359 21929 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:00.415673 21929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497: Generated new TSK 0
I20260928 22:03:00.415889 21929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.165.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
Entry for principal kudu/127.2.165.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab.
I20260928 22:03:00.448287 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:37285
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:00.536556 21933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:00.536803 21933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:00.536831 21933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:00.538153 21933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:00.538206 21933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:00.538234 21933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:03:00.539773 21933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.65.keytab
--principal=kudu/127.2.165.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:37285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:00.540051 21933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:00.540293 21933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:00.546283 21941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:00.546343 21939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:00.546306 21938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:00.547004 21933 server_base.cc:1034] running on GCE node
I20260928 22:03:00.547278 21933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:00.547715 21933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:00.548864 21933 hybrid_clock.cc:648] HybridClock initialized: now 1790632980548839 us; error 47 us; skew 500 ppm
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:00.551883 21933 init.cc:377] Logged in from keytab as kudu/127.2.165.65@KRBTEST.COM (short username kudu)
I20260928 22:03:00.552418 21933 webserver.cc:460] Webserver started at http://127.2.165.65:42193/ using document root <none> and password file <none>
I20260928 22:03:00.552708 21933 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:00.552796 21933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:00.552958 21933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:00.554144 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "40024e6e760045e2a8d5aa62b47cdd2b"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "5ef3d86ea4035d60f7f68a3dc88895dc"
server_key_iv: "3e9d08a99319b7e290092262097fe082"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.554524 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "40024e6e760045e2a8d5aa62b47cdd2b"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "5ef3d86ea4035d60f7f68a3dc88895dc"
server_key_iv: "3e9d08a99319b7e290092262097fe082"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.555954 21933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:03:00.556871 21948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.557062 21933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:00.557173 21933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "40024e6e760045e2a8d5aa62b47cdd2b"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "5ef3d86ea4035d60f7f68a3dc88895dc"
server_key_iv: "3e9d08a99319b7e290092262097fe082"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.557267 21933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:00.574190 21933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:00.576542 21933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:00.576814 21933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:00.577086 21933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:03:00.577414 21933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:03:00.577466 21933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.577497 21933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:03:00.577515 21933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.583575 21933 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:36601
I20260928 22:03:00.583621 22061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:36601 every 8 connection(s)
I20260928 22:03:00.584018 21933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.65@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:00.593950 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 21933
I20260928 22:03:00.594091 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:03:00.594319 2709 external_mini_cluster.cc:1428] Setting key 74d9f2448e29774adddca017e2a2bff6
I20260928 22:03:00.594373 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.584795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:60229 (local address 127.2.165.126:37285)
0928 22:03:00.584873 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.584875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.585122 (+ 247us) server_negotiation.cc:407] Connection header received
0928 22:03:00.585600 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.585601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.585612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.585680 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:00.586809 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.587321 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.588360 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.588565 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.588626 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.591199 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:00.591201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:00.591202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:00.591210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:00.592812 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.593285 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.593287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.593288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.593354 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.593641 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.593642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.593643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.593793 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:00.593865 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.594211 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.594303 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:03:00.594928 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.584779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:37285 (local address 127.2.165.65:60229)
0928 22:03:00.584964 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:00.585009 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:00.585507 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:00.585788 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:00.585792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:00.586416 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:00.586687 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.586693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.587432 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.587432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.588268 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.588270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.588650 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.588651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.588857 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.589439 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:00.589449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:00.591085 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:00.592922 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:00.592932 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:00.592935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:00.593199 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:00.593432 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:00.593432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:00.593433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:00.593564 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:00.593867 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:00.593875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:00.594040 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 22:03:00.594495 (+ 455us) client_negotiation.cc:240] Negotiation successful
0928 22:03:00.594707 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260928 22:03:00.595500 22062 heartbeater.cc:347] Connected to a master server at 127.2.165.126:37285
I20260928 22:03:00.595625 22062 heartbeater.cc:464] Registering TS with master...
I20260928 22:03:00.595841 22062 heartbeater.cc:511] Master 127.2.165.126:37285 requested a full tablet report, sending...
I20260928 22:03:00.596457 21871 ts_manager.cc:194] Registered new tserver with Master: 40024e6e760045e2a8d5aa62b47cdd2b (127.2.165.65:36601)
I20260928 22:03:00.597235 21871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.65@KRBTEST.COM'} at 127.2.165.65:60229
WARNING: no policy specified for kudu/127.2.165.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
Entry for principal kudu/127.2.165.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab.
I20260928 22:03:00.624565 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:37285
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:00.706923 22067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:00.707157 22067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:00.707183 22067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:00.708292 22067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:00.708329 22067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:00.708355 22067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:03:00.709794 22067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.66.keytab
--principal=kudu/127.2.165.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:37285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:00.710003 22067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:00.710191 22067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:00.715421 22073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:00.715442 22072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:00.715596 22067 server_base.cc:1034] running on GCE node
W20260928 22:03:00.715421 22075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:00.715878 22067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:00.716102 22067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:00.717267 22067 hybrid_clock.cc:648] HybridClock initialized: now 1790632980717249 us; error 33 us; skew 500 ppm
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:00.719998 22067 init.cc:377] Logged in from keytab as kudu/127.2.165.66@KRBTEST.COM (short username kudu)
I20260928 22:03:00.720528 22067 webserver.cc:460] Webserver started at http://127.2.165.66:45509/ using document root <none> and password file <none>
I20260928 22:03:00.720839 22067 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:00.720911 22067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:00.721055 22067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:00.722193 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "f7d04c36c97a4f969f141cc1a948e9c9"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "a8c79ec218122526e7d9d6adb2cb46a4"
server_key_iv: "d5c6c84d0176d449f4d3f04b6fc6c6ee"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.722575 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "f7d04c36c97a4f969f141cc1a948e9c9"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "a8c79ec218122526e7d9d6adb2cb46a4"
server_key_iv: "d5c6c84d0176d449f4d3f04b6fc6c6ee"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.723973 22067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:03:00.725041 22082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.725241 22067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:00.725346 22067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "f7d04c36c97a4f969f141cc1a948e9c9"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "a8c79ec218122526e7d9d6adb2cb46a4"
server_key_iv: "d5c6c84d0176d449f4d3f04b6fc6c6ee"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.725421 22067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:00.742614 22067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:00.745038 22067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:00.745234 22067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:00.745483 22067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:03:00.745841 22067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:03:00.745905 22067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.745962 22067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:03:00.745983 22067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.754696 22067 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:33001
I20260928 22:03:00.754721 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:33001 every 8 connection(s)
I20260928 22:03:00.755239 22067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:03:00.759837 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 22067
I20260928 22:03:00.759954 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:03:00.760195 2709 external_mini_cluster.cc:1428] Setting key 82edb4e832380f0ccdf3fc8798e16c8e
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.66@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:00.767725 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.756488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:55415 (local address 127.2.165.126:37285)
0928 22:03:00.756556 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.756559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.756573 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:03:00.757180 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.757181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.757195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.757242 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:00.758416 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.759367 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.761064 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.761286 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.761322 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.764013 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:00.764015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:00.764016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:00.764024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:00.765760 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.766247 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.766249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.766249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.766310 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.766637 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.766638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.766639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.766800 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:00.766850 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.767444 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.767558 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:03:00.767774 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.756131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:37285 (local address 127.2.165.66:55415)
0928 22:03:00.756327 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:00.756377 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:00.757070 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:00.757334 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:00.757338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:00.758035 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:00.758284 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.758292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.759475 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.759476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.760952 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.760956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.761396 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.761397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.761507 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.762084 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:00.762097 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:00.763902 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:00.765868 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:00.765876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:00.765879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:00.766130 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:00.766391 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:00.766392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:00.766392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:00.766552 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:00.766903 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:00.766911 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:00.767118 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 22:03:00.767262 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 22:03:00.767470 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260928 22:03:00.768409 22196 heartbeater.cc:347] Connected to a master server at 127.2.165.126:37285
I20260928 22:03:00.768536 22196 heartbeater.cc:464] Registering TS with master...
I20260928 22:03:00.768812 22196 heartbeater.cc:511] Master 127.2.165.126:37285 requested a full tablet report, sending...
I20260928 22:03:00.769304 21871 ts_manager.cc:194] Registered new tserver with Master: f7d04c36c97a4f969f141cc1a948e9c9 (127.2.165.66:33001)
I20260928 22:03:00.769851 21871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.66@KRBTEST.COM'} at 127.2.165.66:55415
WARNING: no policy specified for kudu/127.2.165.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
Entry for principal kudu/127.2.165.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab.
I20260928 22:03:00.788901 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:37285
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:00.872519 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:00.872780 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:00.872807 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:00.874116 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:00.874166 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:00.874200 22201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:03:00.875789 22201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.67.keytab
--principal=kudu/127.2.165.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:37285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:00.876052 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:00.876305 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:00.881419 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:00.881475 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:00.881506 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:00.881914 22201 server_base.cc:1034] running on GCE node
I20260928 22:03:00.882202 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:00.882472 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:00.883638 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790632980883627 us; error 32 us; skew 500 ppm
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:00.886405 22201 init.cc:377] Logged in from keytab as kudu/127.2.165.67@KRBTEST.COM (short username kudu)
I20260928 22:03:00.886878 22201 webserver.cc:460] Webserver started at http://127.2.165.67:41265/ using document root <none> and password file <none>
I20260928 22:03:00.887120 22201 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:00.887199 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:00.887354 22201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:00.888451 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "986efd983875452e95beba07345a1df0"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "05a4be7732cd30bb1510de7387eac6b2"
server_key_iv: "26fda1e4417130c8e5b1f82a3d586677"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.888887 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "986efd983875452e95beba07345a1df0"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "05a4be7732cd30bb1510de7387eac6b2"
server_key_iv: "26fda1e4417130c8e5b1f82a3d586677"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.890337 22201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:00.891206 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.891408 22201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:00.891528 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "986efd983875452e95beba07345a1df0"
format_stamp: "Formatted at 2026-09-28 22:03:00 on dist-test-slave-3t36"
server_key: "05a4be7732cd30bb1510de7387eac6b2"
server_key_iv: "26fda1e4417130c8e5b1f82a3d586677"
server_key_version: "encryptionkey@0"
I20260928 22:03:00.891606 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:00.907148 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:00.909332 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:00.909502 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:00.909744 22201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:03:00.910084 22201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:03:00.910140 22201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.910187 22201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:03:00.910233 22201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:00.916646 22201 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:33377
I20260928 22:03:00.916723 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:33377 every 8 connection(s)
I20260928 22:03:00.917048 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:03:00 dist-test-slave-3t36 krb5kdc[21826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632980, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.67@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:00.924671 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 22201
I20260928 22:03:00.924810 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:03:00.925025 2709 external_mini_cluster.cc:1428] Setting key 2f8e945d18e71a913f3af459adc0ec98
I20260928 22:03:00.926865 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.917821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:57659 (local address 127.2.165.126:37285)
0928 22:03:00.917881 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.917883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.918097 (+ 214us) server_negotiation.cc:407] Connection header received
0928 22:03:00.918622 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.918623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.918635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.918698 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:00.919688 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.920213 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.921294 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.921488 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.921604 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.923766 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:00.923768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:00.923768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:00.923776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:00.925328 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.925768 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.925770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.925770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.925834 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.926136 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.926137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.926138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.926269 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:00.926349 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.926683 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.926773 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:03:00.927100 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.917804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:37285 (local address 127.2.165.67:57659)
0928 22:03:00.917956 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:00.918000 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:00.918505 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:00.918765 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:00.918768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:00.919363 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:00.919582 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.919587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.920305 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.920306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.921212 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.921215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.921613 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.921614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.921718 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.922226 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:00.922235 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:00.923663 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:00.925423 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:00.925432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:00.925434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:00.925676 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:00.925910 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:00.925911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:00.925911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:00.926058 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:00.926339 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:00.926346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:00.926513 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 22:03:00.926632 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:03:00.926809 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 22:03:00.927697 22330 heartbeater.cc:347] Connected to a master server at 127.2.165.126:37285
I20260928 22:03:00.927809 22330 heartbeater.cc:464] Registering TS with master...
I20260928 22:03:00.928058 22330 heartbeater.cc:511] Master 127.2.165.126:37285 requested a full tablet report, sending...
I20260928 22:03:00.928514 21871 ts_manager.cc:194] Registered new tserver with Master: 986efd983875452e95beba07345a1df0 (127.2.165.67:33377)
I20260928 22:03:00.928996 21871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.165.67@KRBTEST.COM'} at 127.2.165.67:57659
I20260928 22:03:00.937083 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:03:00.945230 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.938513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.2.165.126:37285)
0928 22:03:00.938591 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.938593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.938697 (+ 104us) server_negotiation.cc:407] Connection header received
0928 22:03:00.938792 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.938793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.938805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.938878 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:00.939499 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.940001 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.940832 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.941030 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.941118 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.942337 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:00.942339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:00.942339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:00.942347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:00.943825 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.944315 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.944316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.944316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.944379 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:00.944576 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:00.944577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:00.944577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:00.944812 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:00.944923 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.945032 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.945120 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:03:00.946808 21871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:03:00.948553 21871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:03:00.955114 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.950911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:33377 (local address 127.0.0.1:43242)
0928 22:03:00.951287 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:00.951300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:00.951352 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:00.951541 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:00.951542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:00.951544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:00.951581 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.951582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.953005 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.953006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.954001 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.954004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.954885 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.954886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.954944 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.954957 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 22:03:00.955014 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:03:00.955053 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260928 22:03:00.955155 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.951019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.2.165.67:33377)
0928 22:03:00.951333 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.951337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.951347 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:03:00.951434 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.951437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.951452 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.951651 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:00.951724 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.952884 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.954093 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.954778 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.954886 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.954946 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.955027 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.955071 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":85,"threads_started":1}
I20260928 22:03:00.955847 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.950911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:33001 (local address 127.0.0.1:49692)
0928 22:03:00.951072 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:00.951119 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:00.951187 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:00.951467 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:00.951472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:00.951476 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:00.951563 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.951566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.952972 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.952973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.954300 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.954302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.955601 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.955602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.955664 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.955668 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:03:00.955738 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:03:00.955779 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":140,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260928 22:03:00.956024 22264 tablet_service.cc:1467] Processing CreateTablet for tablet e6015aca8d2049f59534c200f669c64d (DEFAULT_TABLE table=test-table [id=6ff8bdc477574386a39a06c8f0f1a295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:03:00.956209 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.951116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.2.165.66:33001)
0928 22:03:00.951281 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.951287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.951297 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:03:00.951338 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.951341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.951358 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.951645 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:00.951724 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.952869 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.954388 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.955128 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.955198 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.955259 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.956085 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.956131 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 22:03:00.956378 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.951122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:36601 (local address 127.0.0.1:53310)
0928 22:03:00.951533 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:00.951543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:00.951595 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:00.951870 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:00.951870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:00.951873 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:00.951927 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.951928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.952856 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.952857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.954295 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:00.954299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.956156 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.956157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:00.956220 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.956224 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:03:00.956280 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:03:00.956320 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":71,"thread_start_us":82,"threads_started":1}
I20260928 22:03:00.956436 22264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6015aca8d2049f59534c200f669c64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:00.956874 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:00.951358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.2.165.65:36601)
0928 22:03:00.951511 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:00.951515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:00.951651 (+ 136us) server_negotiation.cc:407] Connection header received
0928 22:03:00.951697 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:00.951699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:00.951714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:00.951790 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:00.952003 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.952755 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.955296 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:00.956003 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:00.956065 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:00.956123 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:00.956724 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 22:03:00.956771 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260928 22:03:00.956952 22122 tablet_service.cc:1467] Processing CreateTablet for tablet e6015aca8d2049f59534c200f669c64d (DEFAULT_TABLE table=test-table [id=6ff8bdc477574386a39a06c8f0f1a295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:03:00.957295 22122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6015aca8d2049f59534c200f669c64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:00.959501 22347 tablet_bootstrap.cc:492] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Bootstrap starting.
I20260928 22:03:00.959967 22348 tablet_bootstrap.cc:492] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Bootstrap starting.
I20260928 22:03:00.960289 22347 tablet_bootstrap.cc:654] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:00.960430 21996 tablet_service.cc:1467] Processing CreateTablet for tablet e6015aca8d2049f59534c200f669c64d (DEFAULT_TABLE table=test-table [id=6ff8bdc477574386a39a06c8f0f1a295]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:03:00.960711 22348 tablet_bootstrap.cc:654] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:00.960829 21996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6015aca8d2049f59534c200f669c64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:00.960942 22347 log.cc:824] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:00.961351 22348 log.cc:824] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:00.961750 22347 tablet_bootstrap.cc:492] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: No bootstrap required, opened a new log
I20260928 22:03:00.961796 22347 ts_tablet_manager.cc:1397] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:03:00.962150 22348 tablet_bootstrap.cc:492] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: No bootstrap required, opened a new log
I20260928 22:03:00.962284 22348 ts_tablet_manager.cc:1397] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:03:00.963878 22347 raft_consensus.cc:348] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:00.963907 22351 tablet_bootstrap.cc:492] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b: Bootstrap starting.
I20260928 22:03:00.964100 22347 raft_consensus.cc:374] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:00.964159 22347 raft_consensus.cc:729] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986efd983875452e95beba07345a1df0, State: Initialized, Role: FOLLOWER
I20260928 22:03:00.964334 22347 consensus_queue.cc:260] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:00.964359 22348 raft_consensus.cc:348] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:00.964555 22348 raft_consensus.cc:374] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:00.964653 22348 raft_consensus.cc:729] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d04c36c97a4f969f141cc1a948e9c9, State: Initialized, Role: FOLLOWER
I20260928 22:03:00.964713 22351 tablet_bootstrap.cc:654] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:00.964720 22330 heartbeater.cc:503] Master 127.2.165.126:37285 was elected leader, sending a full tablet report...
I20260928 22:03:00.964850 22348 consensus_queue.cc:260] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:00.965152 22348 ts_tablet_manager.cc:1428] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:03:00.965167 22196 heartbeater.cc:503] Master 127.2.165.126:37285 was elected leader, sending a full tablet report...
I20260928 22:03:00.965363 22351 log.cc:824] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:00.964706 22347 ts_tablet_manager.cc:1428] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:03:00.966181 22351 tablet_bootstrap.cc:492] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b: No bootstrap required, opened a new log
I20260928 22:03:00.966277 22351 ts_tablet_manager.cc:1397] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:03:00.967795 22351 raft_consensus.cc:348] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:00.967927 22351 raft_consensus.cc:374] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:00.967948 22351 raft_consensus.cc:729] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40024e6e760045e2a8d5aa62b47cdd2b, State: Initialized, Role: FOLLOWER
I20260928 22:03:00.968080 22351 consensus_queue.cc:260] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:00.968374 22351 ts_tablet_manager.cc:1428] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:03:00.968395 22062 heartbeater.cc:503] Master 127.2.165.126:37285 was elected leader, sending a full tablet report...
W20260928 22:03:01.005935 22197 tablet.cc:2369] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:03:01.084900 22063 tablet.cc:2369] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:03:01.144770 22353 raft_consensus.cc:482] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:03:01.144913 22353 raft_consensus.cc:504] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:01.145359 22353 leader_election.cc:288] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 986efd983875452e95beba07345a1df0 (127.2.165.67:33377), 40024e6e760045e2a8d5aa62b47cdd2b (127.2.165.65:36601)
I20260928 22:03:01.148610 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.145786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:52827 (local address 127.2.165.67:33377)
0928 22:03:01.145875 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.145877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.145887 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:03:01.145920 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.145921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.145932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.146021 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:01.146201 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.146721 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.147741 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.148222 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.148315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.148324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.148492 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.148533 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 22:03:01.148610 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.145647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:33377 (local address 127.2.165.66:52827)
0928 22:03:01.145748 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.145764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.145835 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.146048 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.146049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.146051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:01.146132 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.146134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.146819 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.146821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.147634 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.147636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.148311 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.148312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.148408 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.148411 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.148481 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.148529 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":5}
I20260928 22:03:01.148758 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.145860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:39795 (local address 127.2.165.65:36601)
0928 22:03:01.145908 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.145910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.146219 (+ 309us) server_negotiation.cc:407] Connection header received
0928 22:03:01.146246 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.146247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.146259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.146325 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:01.146476 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.146930 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.147932 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.148337 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.148428 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.148436 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.148601 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.148645 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:03:01.148769 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.145787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:36601 (local address 127.2.165.66:39795)
0928 22:03:01.145957 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.145965 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.146040 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.146328 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.146329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.146331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:01.146395 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.146396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.147018 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.147019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.147824 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.147826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.148428 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.148429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.148504 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.148509 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.148582 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.148627 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
I20260928 22:03:01.149098 22016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6015aca8d2049f59534c200f669c64d" candidate_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" is_pre_election: true
I20260928 22:03:01.149242 22284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6015aca8d2049f59534c200f669c64d" candidate_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986efd983875452e95beba07345a1df0" is_pre_election: true
I20260928 22:03:01.149302 22016 raft_consensus.cc:2454] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7d04c36c97a4f969f141cc1a948e9c9 in term 0.
I20260928 22:03:01.149370 22284 raft_consensus.cc:2454] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7d04c36c97a4f969f141cc1a948e9c9 in term 0.
I20260928 22:03:01.149573 22083 leader_election.cc:302] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40024e6e760045e2a8d5aa62b47cdd2b, f7d04c36c97a4f969f141cc1a948e9c9; no voters:
I20260928 22:03:01.149775 22353 raft_consensus.cc:2781] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:03:01.149855 22353 raft_consensus.cc:482] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:03:01.149886 22353 raft_consensus.cc:3037] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:01.150621 22353 raft_consensus.cc:504] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:01.150784 22353 leader_election.cc:288] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [CANDIDATE]: Term 1 election: Requested vote from peers 986efd983875452e95beba07345a1df0 (127.2.165.67:33377), 40024e6e760045e2a8d5aa62b47cdd2b (127.2.165.65:36601)
I20260928 22:03:01.150966 22284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6015aca8d2049f59534c200f669c64d" candidate_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986efd983875452e95beba07345a1df0"
I20260928 22:03:01.151067 22016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6015aca8d2049f59534c200f669c64d" candidate_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40024e6e760045e2a8d5aa62b47cdd2b"
I20260928 22:03:01.151081 22284 raft_consensus.cc:3037] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:01.151145 22016 raft_consensus.cc:3037] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:01.152057 22284 raft_consensus.cc:2454] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7d04c36c97a4f969f141cc1a948e9c9 in term 1.
I20260928 22:03:01.152057 22016 raft_consensus.cc:2454] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7d04c36c97a4f969f141cc1a948e9c9 in term 1.
I20260928 22:03:01.152283 22083 leader_election.cc:302] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40024e6e760045e2a8d5aa62b47cdd2b, f7d04c36c97a4f969f141cc1a948e9c9; no voters:
I20260928 22:03:01.152400 22353 raft_consensus.cc:2781] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:01.152571 22353 raft_consensus.cc:686] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 1 LEADER]: Becoming Leader. State: Replica: f7d04c36c97a4f969f141cc1a948e9c9, State: Running, Role: LEADER
I20260928 22:03:01.152712 22353 consensus_queue.cc:237] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } }
I20260928 22:03:01.153653 21871 catalog_manager.cc:5495] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7d04c36c97a4f969f141cc1a948e9c9 (127.2.165.66). New cstate: current_term: 1 leader_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d04c36c97a4f969f141cc1a948e9c9" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 } health_report { overall_health: UNKNOWN } } }
W20260928 22:03:01.167630 22331 tablet.cc:2369] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:03:01.182958 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.180402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.2.165.66:33001)
0928 22:03:01.180476 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.180521 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.180529 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:01.180566 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.180567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.180586 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.180674 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:01.180818 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.181301 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.182098 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.182300 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.182384 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.182566 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:01.182681 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:01.182783 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.182803 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.182826 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:03:01.185163 22284 raft_consensus.cc:1261] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 1 FOLLOWER]: Refusing update from remote peer f7d04c36c97a4f969f141cc1a948e9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:01.185220 22016 raft_consensus.cc:1261] T e6015aca8d2049f59534c200f669c64d P 40024e6e760045e2a8d5aa62b47cdd2b [term 1 FOLLOWER]: Refusing update from remote peer f7d04c36c97a4f969f141cc1a948e9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:01.185606 22353 consensus_queue.cc:1035] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "986efd983875452e95beba07345a1df0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:01.185771 22357 consensus_queue.cc:1035] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40024e6e760045e2a8d5aa62b47cdd2b" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:01.188498 22361 mvcc.cc:205] Tried to move back new op lower bound from 7334432690930749440 to 7334432690802167808. Current Snapshot: MvccSnapshot[applied={T|T < 7334432690930749440}]
I20260928 22:03:01.188612 22362 mvcc.cc:205] Tried to move back new op lower bound from 7334432690930749440 to 7334432690802167808. Current Snapshot: MvccSnapshot[applied={T|T < 7334432690930749440}]
I20260928 22:03:01.188920 22360 mvcc.cc:205] Tried to move back new op lower bound from 7334432690930749440 to 7334432690802167808. Current Snapshot: MvccSnapshot[applied={T|T < 7334432690930749440 or (T in {7334432690930749440})}]
I20260928 22:03:01.190433 21870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:03:01.190543 21870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:03:01.191440 21870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 285b19d68e4446a098ccbe6b81195497: Sending DeleteTablet for 3 replicas of tablet e6015aca8d2049f59534c200f669c64d
I20260928 22:03:01.191846 22122 tablet_service.cc:1514] Processing DeleteTablet for tablet e6015aca8d2049f59534c200f669c64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:03:01 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:49692
I20260928 22:03:01.191931 22264 tablet_service.cc:1514] Processing DeleteTablet for tablet e6015aca8d2049f59534c200f669c64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:03:01 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:43242
I20260928 22:03:01.191931 21996 tablet_service.cc:1514] Processing DeleteTablet for tablet e6015aca8d2049f59534c200f669c64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:03:01 UTC) from {username='kudu', principal='kudu/127.2.165.126@KRBTEST.COM'} at 127.0.0.1:53310
I20260928 22:03:01.192147 22370 tablet_replica.cc:330] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: stopping tablet replica
I20260928 22:03:01.192281 22369 tablet_replica.cc:330] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: stopping tablet replica
I20260928 22:03:01.192377 22369 raft_consensus.cc:2229] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:03:01.192474 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 21933
I20260928 22:03:01.192505 22369 raft_consensus.cc:2258] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:03:01.194577 22369 ts_tablet_manager.cc:1905] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:03:01.195144 22370 raft_consensus.cc:2229] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:03:01.195281 22370 raft_consensus.cc:2258] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:03:01.195776 22370 ts_tablet_manager.cc:1905] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:03:01.196717 22369 ts_tablet_manager.cc:1918] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:03:01.196786 22369 log.cc:1196] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/e6015aca8d2049f59534c200f669c64d
I20260928 22:03:01.197055 22369 ts_tablet_manager.cc:1939] T e6015aca8d2049f59534c200f669c64d P f7d04c36c97a4f969f141cc1a948e9c9: Deleting consensus metadata
I20260928 22:03:01.197466 22370 ts_tablet_manager.cc:1918] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:03:01.197487 21858 catalog_manager.cc:4841] TS f7d04c36c97a4f969f141cc1a948e9c9 (127.2.165.66:33001): tablet e6015aca8d2049f59534c200f669c64d (table test-table [id=6ff8bdc477574386a39a06c8f0f1a295]) successfully deleted
I20260928 22:03:01.197530 22370 log.cc:1196] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/e6015aca8d2049f59534c200f669c64d
I20260928 22:03:01.197806 22370 ts_tablet_manager.cc:1939] T e6015aca8d2049f59534c200f669c64d P 986efd983875452e95beba07345a1df0: Deleting consensus metadata
I20260928 22:03:01.198457 21855 catalog_manager.cc:4841] TS 986efd983875452e95beba07345a1df0 (127.2.165.67:33377): tablet e6015aca8d2049f59534c200f669c64d (table test-table [id=6ff8bdc477574386a39a06c8f0f1a295]) successfully deleted
W20260928 22:03:01.199254 21855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:36601 (error 108)
I20260928 22:03:01.199782 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 22067
I20260928 22:03:01.200358 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.199664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:36601 (local address 127.0.0.1:53320)
0928 22:03:01.200271 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.200318 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582}
W20260928 22:03:01.200623 21855 catalog_manager.cc:4584] TS 40024e6e760045e2a8d5aa62b47cdd2b (127.2.165.65:36601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6015aca8d2049f59534c200f669c64d: Network error: Client connection negotiation failed: client connection to 127.2.165.65:36601: connect: Connection refused (error 111)
I20260928 22:03:01.204116 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 22201
I20260928 22:03:01.209649 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 21839
2026-09-28T22:03:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2127 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T22:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:01Z Disabled control of system clock
I20260928 22:03:01.238513 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41191
--webserver_interface=127.2.165.126
--webserver_port=0
--builtin_ntp_servers=127.2.165.84:43333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:41191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:03:01.325472 22376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:01.325732 22376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:01.325755 22376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:01.327068 22376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:01.327126 22376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:01.327138 22376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:01.327152 22376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:01.327164 22376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:01.328727 22376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:41191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:01.329059 22376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:01.329406 22376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:01.334685 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.334718 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.334718 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:01.334915 22376 server_base.cc:1034] running on GCE node
I20260928 22:03:01.335198 22376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:01.335517 22376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:01.336742 22376 hybrid_clock.cc:648] HybridClock initialized: now 1790632981336711 us; error 47 us; skew 500 ppm
I20260928 22:03:01.338079 22376 webserver.cc:460] Webserver started at http://127.2.165.126:32925/ using document root <none> and password file <none>
I20260928 22:03:01.338357 22376 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:01.338428 22376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:01.338598 22376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:01.339694 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "8dd80ab9ad984ee18bfc78c6c3803a43"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "bcf449d65baa1253abc3b18b90b07794"
server_key_iv: "7645932e7400e52819a0accb6a806b72"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.340070 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "8dd80ab9ad984ee18bfc78c6c3803a43"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "bcf449d65baa1253abc3b18b90b07794"
server_key_iv: "7645932e7400e52819a0accb6a806b72"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.341802 22376 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 22:03:01.342717 22390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.342975 22376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:01.343122 22376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "8dd80ab9ad984ee18bfc78c6c3803a43"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "bcf449d65baa1253abc3b18b90b07794"
server_key_iv: "7645932e7400e52819a0accb6a806b72"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.343219 22376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:01.358713 22376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:01.359103 22376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:01.359270 22376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:01.363667 22376 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:41191
I20260928 22:03:01.363703 22442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:41191 every 8 connection(s)
I20260928 22:03:01.364225 22376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:01.365367 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:01.367813 22443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43: Bootstrap starting.
I20260928 22:03:01.368371 22443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:01.368613 22443 log.cc:824] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:01.369365 22443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43: No bootstrap required, opened a new log
I20260928 22:03:01.370934 22443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41191 } }
I20260928 22:03:01.371086 22443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:01.371110 22443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd80ab9ad984ee18bfc78c6c3803a43, State: Initialized, Role: FOLLOWER
I20260928 22:03:01.371264 22443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41191 } }
I20260928 22:03:01.371356 22443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:01.371380 22443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:01.371409 22443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:01.371901 22443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41191 } }
I20260928 22:03:01.371997 22443 leader_election.cc:302] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dd80ab9ad984ee18bfc78c6c3803a43; no voters:
I20260928 22:03:01.372143 22443 leader_election.cc:288] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:01.372247 22446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:01.372424 22446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [term 1 LEADER]: Becoming Leader. State: Replica: 8dd80ab9ad984ee18bfc78c6c3803a43, State: Running, Role: LEADER
I20260928 22:03:01.372458 22443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:01.372550 22446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41191 } }
I20260928 22:03:01.372987 22448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dd80ab9ad984ee18bfc78c6c3803a43. Latest consensus state: current_term: 1 leader_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41191 } } }
I20260928 22:03:01.373082 22447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd80ab9ad984ee18bfc78c6c3803a43" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41191 } } }
I20260928 22:03:01.373203 22448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:01.373214 22447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:01.373672 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 22376
I20260928 22:03:01.373785 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:01.374111 2709 external_mini_cluster.cc:1428] Setting key 96de63fc7180387981e99ba1ba9a5dbe
W20260928 22:03:01.374387 22462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:01.374446 22462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:01.374511 22455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:01.374704 22455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:01.376392 22455 catalog_manager.cc:1347] Generated new cluster ID: 41297323056d44db96bc7147be844d88
I20260928 22:03:01.376451 22455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:01.380941 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.374682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.2.165.126:41191)
0928 22:03:01.374893 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.374903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.375069 (+ 166us) server_negotiation.cc:407] Connection header received
0928 22:03:01.375331 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.375340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.375418 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.375575 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:01.375949 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.377390 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.378478 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.379086 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.379547 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.380045 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:01.380050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:01.380050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:01.380264 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:01.380306 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:01.380320 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:01.380597 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.380628 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.380729 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260928 22:03:01.388922 22455 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:01.389747 22455 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:01.408839 22455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43: Generated new TSK 0
I20260928 22:03:01.409160 22455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:01.414723 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.65:0
--local_ip_for_outbound_sockets=127.2.165.65
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41191
--builtin_ntp_servers=127.2.165.84:43333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:03:01.503458 22467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:01.503741 22467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:01.503782 22467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:01.505149 22467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:01.505225 22467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:01.505270 22467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.65
I20260928 22:03:01.506762 22467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.165.65
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.65
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:01.507062 22467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:01.507393 22467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:01.512817 22475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.512817 22472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.512873 22473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:01.513206 22467 server_base.cc:1034] running on GCE node
I20260928 22:03:01.513428 22467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:01.513716 22467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:01.514895 22467 hybrid_clock.cc:648] HybridClock initialized: now 1790632981514872 us; error 40 us; skew 500 ppm
I20260928 22:03:01.516393 22467 webserver.cc:460] Webserver started at http://127.2.165.65:39079/ using document root <none> and password file <none>
I20260928 22:03:01.516652 22467 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:01.516822 22467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:01.516999 22467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:01.518303 22467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data/instance:
uuid: "646aa2b9592546009ab87a2b37ebe463"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "f310eedb45d8d6f639a2446d5eaccfff"
server_key_iv: "c00a5c5ba1f7c4072c7848b1824c6ad3"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.518743 22467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance:
uuid: "646aa2b9592546009ab87a2b37ebe463"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "f310eedb45d8d6f639a2446d5eaccfff"
server_key_iv: "c00a5c5ba1f7c4072c7848b1824c6ad3"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.520169 22467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:03:01.521062 22481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.521260 22467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:01.521364 22467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/wal
uuid: "646aa2b9592546009ab87a2b37ebe463"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "f310eedb45d8d6f639a2446d5eaccfff"
server_key_iv: "c00a5c5ba1f7c4072c7848b1824c6ad3"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.521453 22467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:01.551227 22467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:01.551590 22467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:01.551761 22467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:01.552031 22467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:03:01.552394 22467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:03:01.552452 22467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.552488 22467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:03:01.552536 22467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.559535 22467 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.65:42875
I20260928 22:03:01.559592 22594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.65:42875 every 8 connection(s)
I20260928 22:03:01.560031 22467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/data/info.pb
I20260928 22:03:01.560953 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 22467
I20260928 22:03:01.561088 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-0/wal/instance
I20260928 22:03:01.561375 2709 external_mini_cluster.cc:1428] Setting key d93ac4f16ff2fcdc13886e477486e5d5
I20260928 22:03:01.562687 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.66:0
--local_ip_for_outbound_sockets=127.2.165.66
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41191
--builtin_ntp_servers=127.2.165.84:43333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:03:01.566520 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.561234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.65:45669 (local address 127.2.165.126:41191)
0928 22:03:01.561294 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.561297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.561608 (+ 311us) server_negotiation.cc:407] Connection header received
0928 22:03:01.562240 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.562242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.562256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.562296 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:01.563270 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.563793 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.564916 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.565147 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.565228 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.566178 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:01.566180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:01.566180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:01.566186 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:01.566204 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:01.566210 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:01.566277 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.566398 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.566433 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:03:01.566771 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.561254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:41191 (local address 127.2.165.65:45669)
0928 22:03:01.561446 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.561497 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.562122 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.562381 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.562385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.562395 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:01.563164 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.563172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.563919 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.563920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.564822 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.564825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.565250 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.565251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.565403 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.566015 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:03:01.566015 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:01.566047 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:03:01.566074 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:01.566290 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:01.566298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:01.566301 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.566388 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.566508 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260928 22:03:01.567421 22595 heartbeater.cc:347] Connected to a master server at 127.2.165.126:41191
I20260928 22:03:01.567539 22595 heartbeater.cc:464] Registering TS with master...
I20260928 22:03:01.567752 22595 heartbeater.cc:511] Master 127.2.165.126:41191 requested a full tablet report, sending...
I20260928 22:03:01.568419 22407 ts_manager.cc:194] Registered new tserver with Master: 646aa2b9592546009ab87a2b37ebe463 (127.2.165.65:42875)
I20260928 22:03:01.569536 22407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.65:45669
W20260928 22:03:01.648166 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:01.648388 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:01.648413 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:01.649667 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:01.649717 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:01.649745 22598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.66
I20260928 22:03:01.651064 22598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.165.66
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.66
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:01.651264 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:01.651516 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:01.656616 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.656710 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.656663 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:01.656745 22598 server_base.cc:1034] running on GCE node
I20260928 22:03:01.657024 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:01.657238 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:01.658418 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790632981658404 us; error 28 us; skew 500 ppm
I20260928 22:03:01.659893 22598 webserver.cc:460] Webserver started at http://127.2.165.66:33063/ using document root <none> and password file <none>
I20260928 22:03:01.660111 22598 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:01.660157 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:01.660254 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:01.661419 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data/instance:
uuid: "bf013d81804a4ec88a1614628a7329fe"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "f792f8c080fbaec08d10eed25be6dd4e"
server_key_iv: "5ea4aa3d19c5238b66dfea56e660690d"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.661743 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance:
uuid: "bf013d81804a4ec88a1614628a7329fe"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "f792f8c080fbaec08d10eed25be6dd4e"
server_key_iv: "5ea4aa3d19c5238b66dfea56e660690d"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.663061 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:03:01.663825 22612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.664044 22598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:01.664135 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal
uuid: "bf013d81804a4ec88a1614628a7329fe"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "f792f8c080fbaec08d10eed25be6dd4e"
server_key_iv: "5ea4aa3d19c5238b66dfea56e660690d"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.664229 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:01.694249 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:01.694602 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:01.694718 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:01.694926 22598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:03:01.695348 22598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:03:01.695385 22598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.695410 22598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:03:01.695421 22598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.701385 22598 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.66:34655
I20260928 22:03:01.701472 22725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.66:34655 every 8 connection(s)
I20260928 22:03:01.701848 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/data/info.pb
I20260928 22:03:01.708174 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.702829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:34329 (local address 127.2.165.126:41191)
0928 22:03:01.702907 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.702910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.703133 (+ 223us) server_negotiation.cc:407] Connection header received
0928 22:03:01.703737 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.703738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.703751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.703806 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:01.704570 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.705110 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.706633 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.706839 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.706922 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.707837 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:01.707838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:01.707839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:01.707844 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:01.707869 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:01.707873 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:01.707935 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.708036 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.708088 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:03:01.708315 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 22598
I20260928 22:03:01.708419 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.702747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:41191 (local address 127.2.165.66:34329)
0928 22:03:01.703006 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.703041 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.703615 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.703879 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.703884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.703895 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:01.704460 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.704473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.705217 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.705219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.706534 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.706537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.706927 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.706929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.707078 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.707670 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:03:01.707671 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:01.707703 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:03:01.707727 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:01.707935 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:01.707942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:01.707946 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.708041 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.708147 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260928 22:03:01.708439 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal/instance
I20260928 22:03:01.708747 2709 external_mini_cluster.cc:1428] Setting key ddb8d2eaaad184eaa73ac4f871ccf764
I20260928 22:03:01.708913 22726 heartbeater.cc:347] Connected to a master server at 127.2.165.126:41191
I20260928 22:03:01.709021 22726 heartbeater.cc:464] Registering TS with master...
I20260928 22:03:01.709228 22726 heartbeater.cc:511] Master 127.2.165.126:41191 requested a full tablet report, sending...
I20260928 22:03:01.709606 22407 ts_manager.cc:194] Registered new tserver with Master: bf013d81804a4ec88a1614628a7329fe (127.2.165.66:34655)
I20260928 22:03:01.709841 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.165.67:0
--local_ip_for_outbound_sockets=127.2.165.67
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41191
--builtin_ntp_servers=127.2.165.84:43333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:03:01.710069 22407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.66:34329
W20260928 22:03:01.795159 22729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:01.795401 22729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:01.795426 22729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:01.796656 22729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:01.796731 22729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:01.796759 22729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.165.67
I20260928 22:03:01.798619 22729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.165.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.165.67
--webserver_port=0
--tserver_master_addrs=127.2.165.126:41191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.165.67
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:01.798889 22729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:01.799196 22729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:01.805049 22735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.805039 22737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:01.805035 22734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:01.805374 22729 server_base.cc:1034] running on GCE node
I20260928 22:03:01.805648 22729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:01.805931 22729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:01.807109 22729 hybrid_clock.cc:648] HybridClock initialized: now 1790632981807087 us; error 32 us; skew 500 ppm
I20260928 22:03:01.808454 22729 webserver.cc:460] Webserver started at http://127.2.165.67:33107/ using document root <none> and password file <none>
I20260928 22:03:01.808828 22729 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:01.808883 22729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:01.809029 22729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:01.810178 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data/instance:
uuid: "7bf28eea49a748b9913225b9bb6bd1c0"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "36bc57c5088b03811164b98662be64bc"
server_key_iv: "07ce9c8f24e5080f76a0477d0471cdb5"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.810560 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance:
uuid: "7bf28eea49a748b9913225b9bb6bd1c0"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "36bc57c5088b03811164b98662be64bc"
server_key_iv: "07ce9c8f24e5080f76a0477d0471cdb5"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.811940 22729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:01.812856 22743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.813042 22729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:03:01.813170 22729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal
uuid: "7bf28eea49a748b9913225b9bb6bd1c0"
format_stamp: "Formatted at 2026-09-28 22:03:01 on dist-test-slave-3t36"
server_key: "36bc57c5088b03811164b98662be64bc"
server_key_iv: "07ce9c8f24e5080f76a0477d0471cdb5"
server_key_version: "encryptionkey@0"
I20260928 22:03:01.813259 22729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:01.837533 22729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:01.837882 22729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:01.838044 22729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:01.838316 22729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:03:01.838698 22729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:03:01.838757 22729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.838791 22729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:03:01.838840 22729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:01.844678 22729 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.67:40949
I20260928 22:03:01.844739 22856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.67:40949 every 8 connection(s)
I20260928 22:03:01.845125 22729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/data/info.pb
I20260928 22:03:01.850858 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.846110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.67:36225 (local address 127.2.165.126:41191)
0928 22:03:01.846178 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.846180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.846424 (+ 244us) server_negotiation.cc:407] Connection header received
0928 22:03:01.846900 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.846902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.846911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.846975 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:01.847683 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.848193 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.849319 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.849540 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.849610 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.850549 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:01.850550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:01.850550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:01.850555 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:01.850571 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:01.850576 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:01.850634 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.850759 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.850792 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:03:01.851023 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.846098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:41191 (local address 127.2.165.67:36225)
0928 22:03:01.846295 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.846345 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.846814 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.847036 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.847040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.847048 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:01.847578 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.847582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.848302 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.848303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.849220 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.849223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.849625 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.849626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.849742 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.850337 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:03:01.850338 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:01.850369 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:03:01.850394 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:01.850647 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:01.850652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:01.850654 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.850716 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.850806 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 22:03:01.851639 22857 heartbeater.cc:347] Connected to a master server at 127.2.165.126:41191
I20260928 22:03:01.851763 22857 heartbeater.cc:464] Registering TS with master...
I20260928 22:03:01.851967 22857 heartbeater.cc:511] Master 127.2.165.126:41191 requested a full tablet report, sending...
I20260928 22:03:01.852458 22407 ts_manager.cc:194] Registered new tserver with Master: 7bf28eea49a748b9913225b9bb6bd1c0 (127.2.165.67:40949)
I20260928 22:03:01.852988 22407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.165.67:36225
I20260928 22:03:01.855149 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 22729
I20260928 22:03:01.855379 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal/instance
I20260928 22:03:01.855624 2709 external_mini_cluster.cc:1428] Setting key 1c967def22a129ab3b4e93ac48944e96
I20260928 22:03:01.856863 2709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:03:01.861786 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.858415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.2.165.126:41191)
0928 22:03:01.858512 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.858524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.858618 (+ 94us) server_negotiation.cc:407] Connection header received
0928 22:03:01.858685 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.858686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.858700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.858757 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:01.858981 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.859497 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.860349 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.860562 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.860649 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.861441 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:01.861443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:01.861443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:01.861449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:01.861472 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:01.861476 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:01.861541 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.861671 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.861704 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 22:03:01.863838 22407 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:03:01.865787 22407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:03:01.873037 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.868479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.66:34655 (local address 127.0.0.1:47762)
0928 22:03:01.868804 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.868851 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.868926 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.869236 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.869239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.869242 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:01.869321 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.869324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.870762 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.870763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.871865 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.871867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.872628 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.872629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.872768 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.872788 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.872865 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.872931 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":96,"threads_started":1}
I20260928 22:03:01.873168 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.868482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:40949 (local address 127.0.0.1:34398)
0928 22:03:01.868946 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.868957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.869012 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.869334 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.869335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.869337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:01.869402 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.869403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.870398 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.870399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.871753 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.871756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.872879 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.872880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.872975 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.872978 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.873052 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.873096 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":101,"threads_started":1}
I20260928 22:03:01.873263 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.868632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.2.165.67:40949)
0928 22:03:01.869064 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.869069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.869086 (+ 17us) server_negotiation.cc:407] Connection header received
0928 22:03:01.869159 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.869162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.869178 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.869255 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:01.869489 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.870276 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.871900 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.872794 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.872861 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.872972 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.873086 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.873141 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":118,"threads_started":1}
I20260928 22:03:01.873411 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.868764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.2.165.66:34655)
0928 22:03:01.868923 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.868930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.868944 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:03:01.869031 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.869034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.869059 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.869144 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:01.869403 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.870659 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.871936 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.872527 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.872624 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.872783 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.873265 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.873316 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260928 22:03:01.874146 22660 tablet_service.cc:1467] Processing CreateTablet for tablet fb9fe58d40bc4f088755f3d932816f9b (DEFAULT_TABLE table=test-table [id=60e47f2096ea4c3f9fe0895b65ec31b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:03:01.874123 22791 tablet_service.cc:1467] Processing CreateTablet for tablet fb9fe58d40bc4f088755f3d932816f9b (DEFAULT_TABLE table=test-table [id=60e47f2096ea4c3f9fe0895b65ec31b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:03:01.874531 22660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9fe58d40bc4f088755f3d932816f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:01.874536 22791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9fe58d40bc4f088755f3d932816f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:01.876993 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.868769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.2.165.65:42875)
0928 22:03:01.868973 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.868979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.869204 (+ 225us) server_negotiation.cc:407] Connection header received
0928 22:03:01.869252 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.869254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.869269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.869338 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:01.869531 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.870720 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.871966 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.872733 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.872809 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.872917 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.876789 (+ 3872us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.876864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260928 22:03:01.877332 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.868487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:42875 (local address 127.0.0.1:43308)
0928 22:03:01.869093 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.869102 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.869145 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.869414 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.869415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.869416 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:01.869452 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.869453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.870808 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.870809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.871861 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.871864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.872956 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.872957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.873062 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.873068 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.873642 (+ 574us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.877245 (+ 3603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":87,"threads_started":1}
I20260928 22:03:01.877564 22871 tablet_bootstrap.cc:492] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Bootstrap starting.
I20260928 22:03:01.878252 22871 tablet_bootstrap.cc:654] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:01.878166 22529 tablet_service.cc:1467] Processing CreateTablet for tablet fb9fe58d40bc4f088755f3d932816f9b (DEFAULT_TABLE table=test-table [id=60e47f2096ea4c3f9fe0895b65ec31b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:03:01.878491 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9fe58d40bc4f088755f3d932816f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:01.878563 22871 log.cc:824] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:01.879345 22871 tablet_bootstrap.cc:492] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: No bootstrap required, opened a new log
I20260928 22:03:01.879405 22871 ts_tablet_manager.cc:1397] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:03:01.881096 22873 tablet_bootstrap.cc:492] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Bootstrap starting.
I20260928 22:03:01.881698 22874 tablet_bootstrap.cc:492] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463: Bootstrap starting.
I20260928 22:03:01.881930 22873 tablet_bootstrap.cc:654] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:01.881980 22871 raft_consensus.cc:348] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.882193 22871 raft_consensus.cc:374] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:01.882241 22871 raft_consensus.cc:729] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf013d81804a4ec88a1614628a7329fe, State: Initialized, Role: FOLLOWER
I20260928 22:03:01.882289 22873 log.cc:824] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:01.882362 22874 tablet_bootstrap.cc:654] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:01.882418 22871 consensus_queue.cc:260] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.882712 22874 log.cc:824] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:01.882746 22871 ts_tablet_manager.cc:1428] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:03:01.882797 22726 heartbeater.cc:503] Master 127.2.165.126:41191 was elected leader, sending a full tablet report...
I20260928 22:03:01.883231 22873 tablet_bootstrap.cc:492] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: No bootstrap required, opened a new log
I20260928 22:03:01.883301 22873 ts_tablet_manager.cc:1397] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:03:01.883548 22876 raft_consensus.cc:482] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:03:01.883613 22876 raft_consensus.cc:504] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.883803 22874 tablet_bootstrap.cc:492] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463: No bootstrap required, opened a new log
I20260928 22:03:01.883929 22876 leader_election.cc:288] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf28eea49a748b9913225b9bb6bd1c0 (127.2.165.67:40949), 646aa2b9592546009ab87a2b37ebe463 (127.2.165.65:42875)
I20260928 22:03:01.883875 22874 ts_tablet_manager.cc:1397] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:03:01.885823 22873 raft_consensus.cc:348] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.885990 22873 raft_consensus.cc:374] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:01.886044 22873 raft_consensus.cc:729] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf28eea49a748b9913225b9bb6bd1c0, State: Initialized, Role: FOLLOWER
I20260928 22:03:01.886178 22873 consensus_queue.cc:260] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.886478 22857 heartbeater.cc:503] Master 127.2.165.126:41191 was elected leader, sending a full tablet report...
I20260928 22:03:01.886816 22874 raft_consensus.cc:348] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.886979 22874 raft_consensus.cc:374] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:01.887012 22874 raft_consensus.cc:729] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646aa2b9592546009ab87a2b37ebe463, State: Initialized, Role: FOLLOWER
I20260928 22:03:01.887151 22874 consensus_queue.cc:260] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.887423 22874 ts_tablet_manager.cc:1428] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:03:01.887462 22595 heartbeater.cc:503] Master 127.2.165.126:41191 was elected leader, sending a full tablet report...
I20260928 22:03:01.888054 22873 ts_tablet_manager.cc:1428] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 22:03:01.888659 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.884551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:36101 (local address 127.2.165.67:40949)
0928 22:03:01.884601 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.884603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.884759 (+ 156us) server_negotiation.cc:407] Connection header received
0928 22:03:01.884858 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.884859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.884870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.884899 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:01.885148 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.885961 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.887733 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.888282 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.888355 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.888363 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.888574 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.888613 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:03:01.888720 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.884450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.67:40949 (local address 127.2.165.66:36101)
0928 22:03:01.884696 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.884708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.884786 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.884994 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.884995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.884997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:01.885060 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.885062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.886103 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.886105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.887353 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.887356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.888396 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.888397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.888469 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.888473 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.888558 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.888602 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
I20260928 22:03:01.889001 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9fe58d40bc4f088755f3d932816f9b" candidate_uuid: "bf013d81804a4ec88a1614628a7329fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" is_pre_election: true
I20260928 22:03:01.889178 22811 raft_consensus.cc:2454] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf013d81804a4ec88a1614628a7329fe in term 0.
I20260928 22:03:01.889394 22616 leader_election.cc:302] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf28eea49a748b9913225b9bb6bd1c0, bf013d81804a4ec88a1614628a7329fe; no voters:
I20260928 22:03:01.889536 22876 raft_consensus.cc:2781] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:03:01.889604 22876 raft_consensus.cc:482] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:03:01.889639 22876 raft_consensus.cc:3037] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:01.890192 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.884157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:42875 (local address 127.2.165.66:53709)
0928 22:03:01.884219 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.884233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:01.884274 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:01.886985 (+ 2711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:01.886986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:01.886988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:03:01.887100 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.887102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.887927 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.887928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.888993 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:01.888996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.889944 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.889945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:01.890018 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.890022 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:03:01.890083 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:03:01.890130 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:03:01.890203 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.886800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.165.66:53709 (local address 127.2.165.65:42875)
0928 22:03:01.886841 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.886843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.886850 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:03:01.886872 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.886873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.886882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.886935 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:03:01.887188 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.887845 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.889109 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.889851 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.889956 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.889969 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.890088 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.890136 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:03:01.890493 22549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9fe58d40bc4f088755f3d932816f9b" candidate_uuid: "bf013d81804a4ec88a1614628a7329fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646aa2b9592546009ab87a2b37ebe463" is_pre_election: true
I20260928 22:03:01.890601 22549 raft_consensus.cc:2454] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf013d81804a4ec88a1614628a7329fe in term 0.
I20260928 22:03:01.890666 22876 raft_consensus.cc:504] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.890828 22876 leader_election.cc:288] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [CANDIDATE]: Term 1 election: Requested vote from peers 7bf28eea49a748b9913225b9bb6bd1c0 (127.2.165.67:40949), 646aa2b9592546009ab87a2b37ebe463 (127.2.165.65:42875)
I20260928 22:03:01.890959 22549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9fe58d40bc4f088755f3d932816f9b" candidate_uuid: "bf013d81804a4ec88a1614628a7329fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646aa2b9592546009ab87a2b37ebe463"
I20260928 22:03:01.891018 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9fe58d40bc4f088755f3d932816f9b" candidate_uuid: "bf013d81804a4ec88a1614628a7329fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf28eea49a748b9913225b9bb6bd1c0"
I20260928 22:03:01.891108 22549 raft_consensus.cc:3037] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:01.891203 22811 raft_consensus.cc:3037] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:01.892086 22549 raft_consensus.cc:2454] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf013d81804a4ec88a1614628a7329fe in term 1.
I20260928 22:03:01.892158 22811 raft_consensus.cc:2454] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf013d81804a4ec88a1614628a7329fe in term 1.
I20260928 22:03:01.892302 22613 leader_election.cc:302] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646aa2b9592546009ab87a2b37ebe463, bf013d81804a4ec88a1614628a7329fe; no voters:
I20260928 22:03:01.892391 22876 raft_consensus.cc:2781] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:01.892544 22876 raft_consensus.cc:686] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 1 LEADER]: Becoming Leader. State: Replica: bf013d81804a4ec88a1614628a7329fe, State: Running, Role: LEADER
I20260928 22:03:01.892691 22876 consensus_queue.cc:237] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } }
I20260928 22:03:01.893741 22407 catalog_manager.cc:5495] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe reported cstate change: term changed from 0 to 1, leader changed from <none> to bf013d81804a4ec88a1614628a7329fe (127.2.165.66). New cstate: current_term: 1 leader_uuid: "bf013d81804a4ec88a1614628a7329fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf013d81804a4ec88a1614628a7329fe" member_type: VOTER last_known_addr { host: "127.2.165.66" port: 34655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 } health_report { overall_health: UNKNOWN } } }
I20260928 22:03:01.901427 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.898781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.2.165.66:34655)
0928 22:03:01.898880 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:01.898882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:01.898891 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:03:01.898923 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:01.898923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:01.898941 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:01.899011 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:01.899207 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.899709 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.900487 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:01.900782 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:01.900842 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:01.901024 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:01.901132 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:01.901217 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:01.901255 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:03:01.901274 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 22:03:01.903506 22549 raft_consensus.cc:1261] T fb9fe58d40bc4f088755f3d932816f9b P 646aa2b9592546009ab87a2b37ebe463 [term 1 FOLLOWER]: Refusing update from remote peer bf013d81804a4ec88a1614628a7329fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:01.903548 22811 raft_consensus.cc:1261] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 1 FOLLOWER]: Refusing update from remote peer bf013d81804a4ec88a1614628a7329fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:01.903878 22876 consensus_queue.cc:1035] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "646aa2b9592546009ab87a2b37ebe463" member_type: VOTER last_known_addr { host: "127.2.165.65" port: 42875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:01.904052 22881 consensus_queue.cc:1035] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf28eea49a748b9913225b9bb6bd1c0" member_type: VOTER last_known_addr { host: "127.2.165.67" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:01.907130 22886 mvcc.cc:205] Tried to move back new op lower bound from 7334432693873758208 to 7334432693833347072. Current Snapshot: MvccSnapshot[applied={T|T < 7334432693873758208}]
I20260928 22:03:01.907255 22885 mvcc.cc:205] Tried to move back new op lower bound from 7334432693873758208 to 7334432693833347072. Current Snapshot: MvccSnapshot[applied={T|T < 7334432693873758208}]
I20260928 22:03:01.908866 22406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:03:01.908962 22406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:03:01.909785 22406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8dd80ab9ad984ee18bfc78c6c3803a43: Sending DeleteTablet for 3 replicas of tablet fb9fe58d40bc4f088755f3d932816f9b
I20260928 22:03:01.910133 22791 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9fe58d40bc4f088755f3d932816f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:03:01 UTC) from {username='slave'} at 127.0.0.1:34398
I20260928 22:03:01.910140 22529 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9fe58d40bc4f088755f3d932816f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:03:01 UTC) from {username='slave'} at 127.0.0.1:43308
I20260928 22:03:01.910347 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 22467
I20260928 22:03:01.910354 22660 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9fe58d40bc4f088755f3d932816f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:03:01 UTC) from {username='slave'} at 127.0.0.1:47762
I20260928 22:03:01.911090 22896 tablet_replica.cc:330] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: stopping tablet replica
I20260928 22:03:01.911222 22896 raft_consensus.cc:2229] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:03:01.911384 22896 raft_consensus.cc:2258] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:03:01.912992 22896 ts_tablet_manager.cc:1905] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:03:01.914080 22895 tablet_replica.cc:330] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: stopping tablet replica
I20260928 22:03:01.914225 22895 raft_consensus.cc:2229] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:03:01.914357 22895 raft_consensus.cc:2258] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:03:01.914834 22895 ts_tablet_manager.cc:1905] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:03:01.915066 22896 ts_tablet_manager.cc:1918] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:03:01.915169 22896 log.cc:1196] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-1/wal/wals/fb9fe58d40bc4f088755f3d932816f9b
I20260928 22:03:01.915750 22896 ts_tablet_manager.cc:1939] T fb9fe58d40bc4f088755f3d932816f9b P bf013d81804a4ec88a1614628a7329fe: Deleting consensus metadata
I20260928 22:03:01.916378 22393 catalog_manager.cc:4841] TS bf013d81804a4ec88a1614628a7329fe (127.2.165.66:34655): tablet fb9fe58d40bc4f088755f3d932816f9b (table test-table [id=60e47f2096ea4c3f9fe0895b65ec31b7]) successfully deleted
I20260928 22:03:01.917220 22895 ts_tablet_manager.cc:1918] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:03:01.917302 22895 log.cc:1196] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Deleting WAL directory at /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632730261496-2709-0/minicluster-data/ts-2/wal/wals/fb9fe58d40bc4f088755f3d932816f9b
I20260928 22:03:01.917554 22895 ts_tablet_manager.cc:1939] T fb9fe58d40bc4f088755f3d932816f9b P 7bf28eea49a748b9913225b9bb6bd1c0: Deleting consensus metadata
I20260928 22:03:01.918318 22394 catalog_manager.cc:4841] TS 7bf28eea49a748b9913225b9bb6bd1c0 (127.2.165.67:40949): tablet fb9fe58d40bc4f088755f3d932816f9b (table test-table [id=60e47f2096ea4c3f9fe0895b65ec31b7]) successfully deleted
W20260928 22:03:01.927966 22391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.165.65:42875 (error 108)
I20260928 22:03:01.928166 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 22598
I20260928 22:03:01.929889 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:01.929715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.65:42875 (local address 127.0.0.1:43318)
0928 22:03:01.929772 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:01.929836 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.65:42875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 22:03:01.930126 22391 catalog_manager.cc:4584] TS 646aa2b9592546009ab87a2b37ebe463 (127.2.165.65:42875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb9fe58d40bc4f088755f3d932816f9b: Network error: Client connection negotiation failed: client connection to 127.2.165.65:42875: connect: Connection refused (error 111)
I20260928 22:03:01.933713 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 22729
I20260928 22:03:01.938563 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 22376
2026-09-28T22:03:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (727 ms)
[----------] 6 tests from EncryptionPolicyTest (9051 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:02 dist-test-slave-3t36 krb5kdc[22900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:02 dist-test-slave-3t36 krb5kdc[22900](info): set up 2 sockets
Sep 28 22:03:02 dist-test-slave-3t36 krb5kdc[22900](info): commencing operation
krb5kdc: starting...
W20260928 22:03:03.005195 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:03 dist-test-slave-3t36 krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:03.132882 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:03.215945 22913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:03.216184 22913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:03.216209 22913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:03.217561 22913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:03.217622 22913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:03.217638 22913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:03.217649 22913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:03.217662 22913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:03.219188 22913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:03.219400 22913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:03.219611 22913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:03.224653 22921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:03.224653 22919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:03.224979 22913 server_base.cc:1034] running on GCE node
W20260928 22:03:03.224749 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:03.225322 22913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:03.225665 22913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:03.226824 22913 hybrid_clock.cc:648] HybridClock initialized: now 1790632983226800 us; error 38 us; skew 500 ppm
Sep 28 22:03:03 dist-test-slave-3t36 krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:03.229526 22913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:03.230047 22913 webserver.cc:460] Webserver started at http://127.0.0.1:38863/ using document root <none> and password file <none>
I20260928 22:03:03.230301 22913 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:03.230348 22913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:03.230495 22913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:03.231617 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "1ad2ec3850ca4e09ab636d745a2de30c"
format_stamp: "Formatted at 2026-09-28 22:03:03 on dist-test-slave-3t36"
server_key: "374f703835ee4e864e2d219218f0479e"
server_key_iv: "45f392cdb7544f251e1d1c11cc33445b"
server_key_version: "encryptionkey@0"
I20260928 22:03:03.232023 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "1ad2ec3850ca4e09ab636d745a2de30c"
format_stamp: "Formatted at 2026-09-28 22:03:03 on dist-test-slave-3t36"
server_key: "374f703835ee4e864e2d219218f0479e"
server_key_iv: "45f392cdb7544f251e1d1c11cc33445b"
server_key_version: "encryptionkey@0"
I20260928 22:03:03.233450 22913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:03.234304 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:03.234508 22913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:03.234617 22913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "1ad2ec3850ca4e09ab636d745a2de30c"
format_stamp: "Formatted at 2026-09-28 22:03:03 on dist-test-slave-3t36"
server_key: "374f703835ee4e864e2d219218f0479e"
server_key_iv: "45f392cdb7544f251e1d1c11cc33445b"
server_key_version: "encryptionkey@0"
I20260928 22:03:03.234694 22913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:03.251055 22913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:03.253280 22913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:03.253464 22913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:03.258046 22913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44023
I20260928 22:03:03.258069 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44023 every 8 connection(s)
I20260928 22:03:03.258497 22913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:03.259479 22981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:03.261881 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c: Bootstrap starting.
I20260928 22:03:03.262439 22981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:03.262712 22981 log.cc:824] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:03.263427 22981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c: No bootstrap required, opened a new log
I20260928 22:03:03.265149 22981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:03:03.265300 22981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:03.265323 22981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad2ec3850ca4e09ab636d745a2de30c, State: Initialized, Role: FOLLOWER
I20260928 22:03:03.265467 22981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:03:03.265551 22981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:03.265573 22981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:03.265602 22981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:03.266122 22981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:03:03.266209 22981 leader_election.cc:302] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ad2ec3850ca4e09ab636d745a2de30c; no voters:
I20260928 22:03:03.266353 22981 leader_election.cc:288] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:03.266439 22984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:03.266630 22981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:03.266647 22984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [term 1 LEADER]: Becoming Leader. State: Replica: 1ad2ec3850ca4e09ab636d745a2de30c, State: Running, Role: LEADER
I20260928 22:03:03.266842 22984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } }
I20260928 22:03:03.267280 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } } }
I20260928 22:03:03.267375 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:03.267421 22986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad2ec3850ca4e09ab636d745a2de30c. Latest consensus state: current_term: 1 leader_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2ec3850ca4e09ab636d745a2de30c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44023 } } }
I20260928 22:03:03.267488 22986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:03.267628 22993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:03.268275 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 22913
I20260928 22:03:03.268388 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:03.268585 2709 external_mini_cluster.cc:1428] Setting key 1d655a121fc464ac64070bb832da6db4
I20260928 22:03:03.268924 22993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:03.270843 22993 catalog_manager.cc:1347] Generated new cluster ID: de225b4adab84ac3be3d930c5ddaf0f1
I20260928 22:03:03.270905 22993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:03:03 dist-test-slave-3t36 krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:03:03.285063 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:03.269304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46089 (local address 127.0.0.1:44023)
0928 22:03:03.272864 (+ 3560us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:03.272874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:03.272909 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:03:03.273027 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:03.273034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:03.273108 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:03.273250 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:03.274127 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:03.276079 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:03.277358 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:03.277980 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:03.278091 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:03.281092 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:03.281097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:03.281098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:03.281121 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:03.283237 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:03.283716 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:03.283719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:03.283720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:03.283843 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:03.284074 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:03.284074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:03.284075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:03.284421 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:03.284584 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:03.284720 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:03:03.284832 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3511,"thread_start_us":78,"threads_started":1}
I20260928 22:03:03.286276 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:03.288738 22993 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:03.289456 22993 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:03.294790 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:03.287844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33577 (local address 127.0.0.1:44023)
0928 22:03:03.287881 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:03.287883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:03.288057 (+ 174us) server_negotiation.cc:407] Connection header received
0928 22:03:03.288104 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:03.288105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:03.288116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:03.288178 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:03.288926 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:03.289402 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:03.290151 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:03.290648 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:03.290745 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:03.292030 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:03.292031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:03.292032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:03.292039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:03.293569 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:03.293974 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:03.293976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:03.293976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:03.294035 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:03.294260 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:03.294261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:03.294262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:03.294380 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:03.294519 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:03.294620 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:03:03.294684 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:03:03.301205 22993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ad2ec3850ca4e09ab636d745a2de30c: Generated new TSK 0
I20260928 22:03:03.301456 22993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:03.308645 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 22913
W20260928 22:03:03.313115 22898 connection.cc:535] client connection to 127.0.0.1:44023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44023): Connection reset by peer (error 104)
2026-09-28T22:03:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:03 dist-test-slave-3t36 krb5kdc[23012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:03 dist-test-slave-3t36 krb5kdc[23012](info): set up 2 sockets
Sep 28 22:03:03 dist-test-slave-3t36 krb5kdc[23012](info): commencing operation
krb5kdc: starting...
W20260928 22:03:04.377853 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:04 dist-test-slave-3t36 krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:04.504390 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:04.587400 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:04.587689 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:04.587733 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:04.589073 23025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:04.589143 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:04.589167 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:04.589190 23025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:04.589217 23025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:04.590790 23025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:04.591095 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:04.591346 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:04.596617 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:04.596771 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:04.596833 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:04.596972 23025 server_base.cc:1034] running on GCE node
I20260928 22:03:04.597245 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:04.597576 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:04.598778 23025 hybrid_clock.cc:648] HybridClock initialized: now 1790632984598790 us; error 58 us; skew 500 ppm
Sep 28 22:03:04 dist-test-slave-3t36 krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:04.601538 23025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:04.602058 23025 webserver.cc:460] Webserver started at http://127.0.0.1:39163/ using document root <none> and password file <none>
I20260928 22:03:04.602308 23025 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:04.602378 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:04.602545 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:04.603647 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "dc4090f78e1b4de893d78cf80dae3191"
format_stamp: "Formatted at 2026-09-28 22:03:04 on dist-test-slave-3t36"
server_key: "bac0897f9a44db0784caf2be368d0725"
server_key_iv: "c251fc6288945db191eccf3f262e3aa7"
server_key_version: "encryptionkey@0"
I20260928 22:03:04.604036 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "dc4090f78e1b4de893d78cf80dae3191"
format_stamp: "Formatted at 2026-09-28 22:03:04 on dist-test-slave-3t36"
server_key: "bac0897f9a44db0784caf2be368d0725"
server_key_iv: "c251fc6288945db191eccf3f262e3aa7"
server_key_version: "encryptionkey@0"
I20260928 22:03:04.605530 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:04.606375 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:04.606606 23025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:04.606724 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "dc4090f78e1b4de893d78cf80dae3191"
format_stamp: "Formatted at 2026-09-28 22:03:04 on dist-test-slave-3t36"
server_key: "bac0897f9a44db0784caf2be368d0725"
server_key_iv: "c251fc6288945db191eccf3f262e3aa7"
server_key_version: "encryptionkey@0"
I20260928 22:03:04.606803 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:04.625627 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:04.628137 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:04.628325 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:04.633057 23025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39453
I20260928 22:03:04.633124 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39453 every 8 connection(s)
I20260928 22:03:04.633518 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:04.634639 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:04.636991 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191: Bootstrap starting.
I20260928 22:03:04.637607 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:04.637899 23093 log.cc:824] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:04.638724 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191: No bootstrap required, opened a new log
I20260928 22:03:04.639832 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23025
I20260928 22:03:04.639963 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:04.640161 2709 external_mini_cluster.cc:1428] Setting key 90eaa355b06ef12daee0d8941ca72d0f
I20260928 22:03:04.640576 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4090f78e1b4de893d78cf80dae3191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260928 22:03:04.640815 23093 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:04.640900 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4090f78e1b4de893d78cf80dae3191, State: Initialized, Role: FOLLOWER
I20260928 22:03:04.641052 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4090f78e1b4de893d78cf80dae3191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260928 22:03:04.641134 23093 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:04.641163 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:04.641209 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:04.642370 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4090f78e1b4de893d78cf80dae3191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260928 22:03:04.642513 23093 leader_election.cc:302] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc4090f78e1b4de893d78cf80dae3191; no voters:
I20260928 22:03:04.642676 23093 leader_election.cc:288] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:04.642736 23098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:04.642906 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:04.642926 23098 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [term 1 LEADER]: Becoming Leader. State: Replica: dc4090f78e1b4de893d78cf80dae3191, State: Running, Role: LEADER
I20260928 22:03:04.643076 23098 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4090f78e1b4de893d78cf80dae3191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } }
I20260928 22:03:04.644296 23100 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc4090f78e1b4de893d78cf80dae3191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4090f78e1b4de893d78cf80dae3191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } } }
I20260928 22:03:04.644384 23100 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:04.644574 23111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:04.645160 23111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:04.644644 23099 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc4090f78e1b4de893d78cf80dae3191. Latest consensus state: current_term: 1 leader_uuid: "dc4090f78e1b4de893d78cf80dae3191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4090f78e1b4de893d78cf80dae3191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39453 } } }
I20260928 22:03:04.645239 23099 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:04.646488 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:04.640774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58907 (local address 127.0.0.1:39453)
0928 22:03:04.640987 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:04.640996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:04.641031 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:03:04.641161 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:04.641168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:04.641228 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:04.641390 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:04.641575 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:04.643032 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:04.644150 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:04.644855 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:04.644973 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:04.645712 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:04.645717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:04.645718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:04.645950 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:04.645988 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:04.646003 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:04.646094 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:04.646235 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 22:03:04.646265 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260928 22:03:04.646872 23111 catalog_manager.cc:1347] Generated new cluster ID: 2ef9cdb291b94d879c994c93b5bb1147
I20260928 22:03:04.646924 23111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:04.647466 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:04.655608 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:04.648862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55573 (local address 127.0.0.1:39453)
0928 22:03:04.652757 (+ 3895us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:04.652758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:04.652766 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:04.652786 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:04.652787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:04.652795 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:04.652870 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:04.653064 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:04.653551 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:04.654303 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:04.654518 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:04.654603 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:04.655310 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:04.655311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:04.655311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:04.655316 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:04.655330 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:04.655334 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:04.655387 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:04.655445 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 22:03:04.655456 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3871}
I20260928 22:03:04.665335 23111 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:04.666121 23111 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:04.675029 23111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc4090f78e1b4de893d78cf80dae3191: Generated new TSK 0
I20260928 22:03:04.675325 23111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:04.679761 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23025
W20260928 22:03:04.684193 23010 connection.cc:535] client connection to 127.0.0.1:39453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39453): Connection reset by peer (error 104)
2026-09-28T22:03:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:04 dist-test-slave-3t36 krb5kdc[23124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:04 dist-test-slave-3t36 krb5kdc[23124](info): set up 2 sockets
Sep 28 22:03:04 dist-test-slave-3t36 krb5kdc[23124](info): commencing operation
krb5kdc: starting...
W20260928 22:03:05.712769 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:05 dist-test-slave-3t36 krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:05.837180 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:05.920405 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:05.920651 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:05.920704 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:05.922056 23137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:05.922106 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:05.922119 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:05.922155 23137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:05.922185 23137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:05.923758 23137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:05.924036 23137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:05.924283 23137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:05.929555 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:05.929574 23145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:05.929576 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:05.929979 23137 server_base.cc:1034] running on GCE node
I20260928 22:03:05.930195 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:05.930511 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:05.931716 23137 hybrid_clock.cc:648] HybridClock initialized: now 1790632985931698 us; error 30 us; skew 500 ppm
Sep 28 22:03:05 dist-test-slave-3t36 krb5kdc[23124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:05.934362 23137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:05.934829 23137 webserver.cc:460] Webserver started at http://127.0.0.1:33145/ using document root <none> and password file <none>
I20260928 22:03:05.935060 23137 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:05.935106 23137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:05.935240 23137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:05.936353 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "ed2d6649061b49d3b6d65c1fe64138c6"
format_stamp: "Formatted at 2026-09-28 22:03:05 on dist-test-slave-3t36"
server_key: "ce32b907ce7c92b2dec66820c023f7ee"
server_key_iv: "f1e773aec1f1a3c47f1187cb30ea7be0"
server_key_version: "encryptionkey@0"
I20260928 22:03:05.936785 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "ed2d6649061b49d3b6d65c1fe64138c6"
format_stamp: "Formatted at 2026-09-28 22:03:05 on dist-test-slave-3t36"
server_key: "ce32b907ce7c92b2dec66820c023f7ee"
server_key_iv: "f1e773aec1f1a3c47f1187cb30ea7be0"
server_key_version: "encryptionkey@0"
I20260928 22:03:05.938175 23137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:05.938987 23152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:05.939173 23137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:05.939267 23137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "ed2d6649061b49d3b6d65c1fe64138c6"
format_stamp: "Formatted at 2026-09-28 22:03:05 on dist-test-slave-3t36"
server_key: "ce32b907ce7c92b2dec66820c023f7ee"
server_key_iv: "f1e773aec1f1a3c47f1187cb30ea7be0"
server_key_version: "encryptionkey@0"
I20260928 22:03:05.939328 23137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:05.942832 23137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:05.945147 23137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:05.945254 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:05.949162 23137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43217
I20260928 22:03:05.949226 23204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43217 every 8 connection(s)
I20260928 22:03:05.949576 23137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:05.950690 23205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:05.952280 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23137
I20260928 22:03:05.952450 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:05.952656 2709 external_mini_cluster.cc:1428] Setting key e418932de456b898f4ec420aea09ddc4
I20260928 22:03:05.953595 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6: Bootstrap starting.
I20260928 22:03:05.954348 23205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:05.954705 23205 log.cc:824] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:05.955534 23205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6: No bootstrap required, opened a new log
I20260928 22:03:05.955763 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:05.953337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46391 (local address 127.0.0.1:43217)
0928 22:03:05.953555 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:05.953565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:05.953601 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:03:05.954192 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:05.954201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:05.954239 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:05.954411 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:05.955252 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:05.955255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:05.955255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:05.955259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:05.955297 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:05.955308 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:05.955367 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:05.955457 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 22:03:05.955525 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260928 22:03:05.956892 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:05.958406 23205 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260928 22:03:05.958621 23205 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:05.958663 23205 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2d6649061b49d3b6d65c1fe64138c6, State: Initialized, Role: FOLLOWER
I20260928 22:03:05.958801 23205 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260928 22:03:05.958909 23205 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:05.958959 23205 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:05.959018 23205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:05.959647 23205 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260928 22:03:05.959743 23205 leader_election.cc:302] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2d6649061b49d3b6d65c1fe64138c6; no voters:
I20260928 22:03:05.959942 23205 leader_election.cc:288] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:05.960004 23215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:05.960151 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:05.958814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48311 (local address 127.0.0.1:43217)
0928 22:03:05.958868 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:05.958870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:05.958968 (+ 98us) server_negotiation.cc:407] Connection header received
0928 22:03:05.959053 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:05.959054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:05.959061 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:05.959132 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:05.959901 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:05.959903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:05.959903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:05.959908 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:05.959924 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:05.959929 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:05.959996 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:05.960064 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 22:03:05.960079 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:03:05.960255 23215 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [term 1 LEADER]: Becoming Leader. State: Replica: ed2d6649061b49d3b6d65c1fe64138c6, State: Running, Role: LEADER
I20260928 22:03:05.960278 23205 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:05.960388 23215 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } }
I20260928 22:03:05.960904 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2d6649061b49d3b6d65c1fe64138c6. Latest consensus state: current_term: 1 leader_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } } }
I20260928 22:03:05.960904 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2d6649061b49d3b6d65c1fe64138c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43217 } } }
I20260928 22:03:05.961129 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:05.961354 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:05.961369 23223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:05.962026 23223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:05.963704 23223 catalog_manager.cc:1347] Generated new cluster ID: cd0ad4321c9f4c6ba8aa9f257e83ebad
I20260928 22:03:05.963763 23223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:05.983136 23223 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:05.983377 23223 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:05.989266 23223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2d6649061b49d3b6d65c1fe64138c6: Generated new TSK 0
I20260928 22:03:05.989496 23223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:05.991398 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23137
2026-09-28T22:03:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:06 dist-test-slave-3t36 krb5kdc[23236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:06 dist-test-slave-3t36 krb5kdc[23236](info): set up 2 sockets
Sep 28 22:03:06 dist-test-slave-3t36 krb5kdc[23236](info): commencing operation
krb5kdc: starting...
W20260928 22:03:07.044945 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:07 dist-test-slave-3t36 krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:07.169350 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:07.253819 23249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:07.254061 23249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:07.254086 23249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:07.255295 23249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:07.255343 23249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:07.255358 23249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:07.255370 23249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:07.255383 23249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:07.256996 23249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:07.257274 23249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:07.257517 23249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:07.262574 23257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:07.262704 23255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:07.262704 23254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:07.263285 23249 server_base.cc:1034] running on GCE node
I20260928 22:03:07.263550 23249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:07.263883 23249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:07.265069 23249 hybrid_clock.cc:648] HybridClock initialized: now 1790632987265051 us; error 38 us; skew 500 ppm
Sep 28 22:03:07 dist-test-slave-3t36 krb5kdc[23236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:07.267673 23249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:07.268108 23249 webserver.cc:460] Webserver started at http://127.0.0.1:41029/ using document root <none> and password file <none>
I20260928 22:03:07.268352 23249 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:07.268400 23249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:07.268524 23249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:07.269699 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "85f87dabb09d45edb4f07a0f237a66e1"
format_stamp: "Formatted at 2026-09-28 22:03:07 on dist-test-slave-3t36"
server_key: "da0efca8e1e4918a2da42087c737088b"
server_key_iv: "406d1126b2b7ea3b681160971c5013a1"
server_key_version: "encryptionkey@0"
I20260928 22:03:07.270082 23249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "85f87dabb09d45edb4f07a0f237a66e1"
format_stamp: "Formatted at 2026-09-28 22:03:07 on dist-test-slave-3t36"
server_key: "da0efca8e1e4918a2da42087c737088b"
server_key_iv: "406d1126b2b7ea3b681160971c5013a1"
server_key_version: "encryptionkey@0"
I20260928 22:03:07.271512 23249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:03:07.272610 23264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:07.273092 23249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:07.273176 23249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "85f87dabb09d45edb4f07a0f237a66e1"
format_stamp: "Formatted at 2026-09-28 22:03:07 on dist-test-slave-3t36"
server_key: "da0efca8e1e4918a2da42087c737088b"
server_key_iv: "406d1126b2b7ea3b681160971c5013a1"
server_key_version: "encryptionkey@0"
I20260928 22:03:07.273285 23249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:07.287886 23249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:07.290177 23249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:07.290321 23249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:07.294520 23249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46137
I20260928 22:03:07.294597 23316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46137 every 8 connection(s)
I20260928 22:03:07.294958 23249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:07.295943 23317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:07.298257 23317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1: Bootstrap starting.
I20260928 22:03:07.298867 23317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:07.299170 23317 log.cc:824] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:07.300007 23317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1: No bootstrap required, opened a new log
I20260928 22:03:07.301765 23317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f87dabb09d45edb4f07a0f237a66e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 22:03:07.301913 23317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:07.301934 23317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f87dabb09d45edb4f07a0f237a66e1, State: Initialized, Role: FOLLOWER
I20260928 22:03:07.302114 23317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f87dabb09d45edb4f07a0f237a66e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 22:03:07.302181 23317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:07.302202 23317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:07.302253 23317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:07.302825 23317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f87dabb09d45edb4f07a0f237a66e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 22:03:07.302973 23317 leader_election.cc:302] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f87dabb09d45edb4f07a0f237a66e1; no voters:
I20260928 22:03:07.303205 23317 leader_election.cc:288] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:07.303260 23320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:07.303491 23317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:07.303506 23320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [term 1 LEADER]: Becoming Leader. State: Replica: 85f87dabb09d45edb4f07a0f237a66e1, State: Running, Role: LEADER
I20260928 22:03:07.303772 23320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f87dabb09d45edb4f07a0f237a66e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 22:03:07.304198 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85f87dabb09d45edb4f07a0f237a66e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f87dabb09d45edb4f07a0f237a66e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } } }
I20260928 22:03:07.304179 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f87dabb09d45edb4f07a0f237a66e1. Latest consensus state: current_term: 1 leader_uuid: "85f87dabb09d45edb4f07a0f237a66e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f87dabb09d45edb4f07a0f237a66e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } } }
I20260928 22:03:07.304307 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:07.304385 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:07.304792 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23249
I20260928 22:03:07.304889 23324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:07.304916 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:07.305124 2709 external_mini_cluster.cc:1428] Setting key f024d682cbcebba0078e0aaded1d22a1
I20260928 22:03:07.305505 23324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:07.307272 23324 catalog_manager.cc:1347] Generated new cluster ID: ee8460ae29b04749bf2e6355c9c8937a
I20260928 22:03:07.307334 23324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:03:07 dist-test-slave-3t36 krb5kdc[23236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:03:07.322345 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:07.305806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57035 (local address 127.0.0.1:46137)
0928 22:03:07.306257 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:07.306267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:07.306299 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:03:07.306414 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:07.306422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:07.306476 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:07.306617 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:07.311299 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:07.313221 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:07.314277 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:07.314953 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:07.315076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:07.318055 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:07.318061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:07.318061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:07.318075 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:07.320106 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:07.320545 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:07.320549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:07.320549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:07.320710 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:07.321085 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:07.321086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:07.321087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:07.321433 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:07.321563 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:07.321799 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 22:03:07.322075 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":95,"threads_started":1}
I20260928 22:03:07.323434 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:07.323879 23324 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:07.324386 23324 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:07.332728 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:07.324984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52179 (local address 127.0.0.1:46137)
0928 22:03:07.325023 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:07.325025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:07.325197 (+ 172us) server_negotiation.cc:407] Connection header received
0928 22:03:07.325245 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:07.325246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:07.325255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:07.325307 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:07.325924 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:07.326647 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:07.327822 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:07.328136 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:07.328200 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:07.329761 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:07.329762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:07.329762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:07.329770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:07.331489 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:07.331892 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:07.331893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:07.331893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:07.331950 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:07.332165 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:07.332166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:07.332166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:07.332287 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:07.332390 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:07.332507 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:03:07.332588 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:03:07.334378 23324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85f87dabb09d45edb4f07a0f237a66e1: Generated new TSK 0
I20260928 22:03:07.334585 23324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:07.338490 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23249
2026-09-28T22:03:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:07 dist-test-slave-3t36 krb5kdc[23348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:07 dist-test-slave-3t36 krb5kdc[23348](info): set up 2 sockets
Sep 28 22:03:07 dist-test-slave-3t36 krb5kdc[23348](info): commencing operation
krb5kdc: starting...
W20260928 22:03:08.400826 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:08 dist-test-slave-3t36 krb5kdc[23348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:08.536432 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:08.620491 23361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:08.620791 23361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:08.620836 23361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:08.622023 23361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:08.622102 23361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:08.622130 23361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:08.622162 23361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:08.622189 23361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:08.623723 23361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:08.624017 23361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:08.624264 23361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:08.629806 23367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:08.629789 23369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:08.629935 23366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:08.630215 23361 server_base.cc:1034] running on GCE node
I20260928 22:03:08.630506 23361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:08.630931 23361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:08.632131 23361 hybrid_clock.cc:648] HybridClock initialized: now 1790632988632109 us; error 47 us; skew 500 ppm
Sep 28 22:03:08 dist-test-slave-3t36 krb5kdc[23348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:08.643400 23361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:08.644003 23361 webserver.cc:460] Webserver started at http://127.0.0.1:42005/ using document root <none> and password file <none>
I20260928 22:03:08.644294 23361 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:08.644368 23361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:08.644524 23361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:08.645748 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "5073dc0786ac483d9a198dd3edb46f5a"
format_stamp: "Formatted at 2026-09-28 22:03:08 on dist-test-slave-3t36"
server_key: "c0e21de01da3e3e02864c89eb5807eb4"
server_key_iv: "cc0493d37a458f091594d6f58388990b"
server_key_version: "encryptionkey@0"
I20260928 22:03:08.646150 23361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "5073dc0786ac483d9a198dd3edb46f5a"
format_stamp: "Formatted at 2026-09-28 22:03:08 on dist-test-slave-3t36"
server_key: "c0e21de01da3e3e02864c89eb5807eb4"
server_key_iv: "cc0493d37a458f091594d6f58388990b"
server_key_version: "encryptionkey@0"
I20260928 22:03:08.647617 23361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:03:08.648569 23376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:08.648830 23361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:08.648953 23361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "5073dc0786ac483d9a198dd3edb46f5a"
format_stamp: "Formatted at 2026-09-28 22:03:08 on dist-test-slave-3t36"
server_key: "c0e21de01da3e3e02864c89eb5807eb4"
server_key_iv: "cc0493d37a458f091594d6f58388990b"
server_key_version: "encryptionkey@0"
I20260928 22:03:08.649039 23361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:08.679488 23361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:08.682071 23361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:08.682284 23361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:08.686964 23361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40347
I20260928 22:03:08.686995 23428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40347 every 8 connection(s)
I20260928 22:03:08.687378 23361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:08.688452 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:08.690783 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a: Bootstrap starting.
I20260928 22:03:08.691324 23429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:08.691555 23429 log.cc:824] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:08.692365 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a: No bootstrap required, opened a new log
I20260928 22:03:08.692477 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23361
I20260928 22:03:08.692615 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:08.692819 2709 external_mini_cluster.cc:1428] Setting key eac837ca3789c9ca024ee2b49faa549e
I20260928 22:03:08.694478 23429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5073dc0786ac483d9a198dd3edb46f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260928 22:03:08.694650 23429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:08.694705 23429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5073dc0786ac483d9a198dd3edb46f5a, State: Initialized, Role: FOLLOWER
I20260928 22:03:08.694845 23429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5073dc0786ac483d9a198dd3edb46f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260928 22:03:08.694943 23429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:08.694983 23429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:08.695040 23429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:08.695631 23429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5073dc0786ac483d9a198dd3edb46f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260928 22:03:08.695758 23429 leader_election.cc:302] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5073dc0786ac483d9a198dd3edb46f5a; no voters:
I20260928 22:03:08.695941 23429 leader_election.cc:288] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:08.695977 23434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:08.696139 23434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [term 1 LEADER]: Becoming Leader. State: Replica: 5073dc0786ac483d9a198dd3edb46f5a, State: Running, Role: LEADER
I20260928 22:03:08.696223 23429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:08.696224 23434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5073dc0786ac483d9a198dd3edb46f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } }
I20260928 22:03:08.698015 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5073dc0786ac483d9a198dd3edb46f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5073dc0786ac483d9a198dd3edb46f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } } }
I20260928 22:03:08.698727 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [sys.catalog]: This master's current role is: LEADER
W20260928 22:03:08.698216 23449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:08.698843 23449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:08.699060 23451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:08.698699 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5073dc0786ac483d9a198dd3edb46f5a. Latest consensus state: current_term: 1 leader_uuid: "5073dc0786ac483d9a198dd3edb46f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5073dc0786ac483d9a198dd3edb46f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40347 } } }
I20260928 22:03:08.699141 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:08.699250 23451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:08.699988 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:08.693408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51833 (local address 127.0.0.1:40347)
0928 22:03:08.693627 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:08.693637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:08.693676 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:03:08.694490 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:08.694498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:08.694562 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:08.694739 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:08.694995 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:08.696599 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:08.697871 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:08.698387 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:08.698475 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:08.699285 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:08.699290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:08.699290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:08.699507 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:08.699538 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:08.699550 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:08.699632 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:08.699708 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 22:03:08.699776 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":124,"threads_started":1}
I20260928 22:03:08.701032 23451 catalog_manager.cc:1347] Generated new cluster ID: 650cc46422a34b1f9a2deea94e37207f
I20260928 22:03:08.701085 23451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:08.701079 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:08.706617 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:08.702795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57781 (local address 127.0.0.1:40347)
0928 22:03:08.702856 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:08.702858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:08.703133 (+ 275us) server_negotiation.cc:407] Connection header received
0928 22:03:08.703199 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:08.703199 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:08.703221 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:08.703283 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:08.703446 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:08.704139 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:08.705289 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:08.705570 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:08.705632 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:08.706358 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:08.706360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:08.706360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:08.706365 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:08.706379 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:08.706383 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:08.706432 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:08.706512 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 22:03:08.706535 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:03:08.724462 23451 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:08.725116 23451 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:08.728139 23451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5073dc0786ac483d9a198dd3edb46f5a: Generated new TSK 0
I20260928 22:03:08.728374 23451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:08.735998 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23361
2026-09-28T22:03:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:08 dist-test-slave-3t36 krb5kdc[23460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:08 dist-test-slave-3t36 krb5kdc[23460](info): set up 2 sockets
Sep 28 22:03:08 dist-test-slave-3t36 krb5kdc[23460](info): commencing operation
krb5kdc: starting...
W20260928 22:03:09.772980 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:09 dist-test-slave-3t36 krb5kdc[23460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:09.900185 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:09.986219 23473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:09.986491 23473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:09.986532 23473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:09.987798 23473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:09.987871 23473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:09.987895 23473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:09.987927 23473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:09.987974 23473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:09.989704 23473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:09.990020 23473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:09.990264 23473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:03:09.995754 23473 server_base.cc:1034] running on GCE node
W20260928 22:03:09.995720 23481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:09.995728 23478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:09.995720 23479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:09.996186 23473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:09.996446 23473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:09.997629 23473 hybrid_clock.cc:648] HybridClock initialized: now 1790632989997604 us; error 46 us; skew 500 ppm
Sep 28 22:03:09 dist-test-slave-3t36 krb5kdc[23460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:10.000551 23473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:10.001114 23473 webserver.cc:460] Webserver started at http://127.0.0.1:35523/ using document root <none> and password file <none>
I20260928 22:03:10.001403 23473 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:10.001478 23473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:10.001652 23473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:10.002813 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "d87dbd51f4d2414bbcc7a22bf6252c84"
format_stamp: "Formatted at 2026-09-28 22:03:09 on dist-test-slave-3t36"
server_key: "81e851203492c34c0d2e1d72fd5baa97"
server_key_iv: "8e774b22134870b4cd07986a09864303"
server_key_version: "encryptionkey@0"
I20260928 22:03:10.003203 23473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "d87dbd51f4d2414bbcc7a22bf6252c84"
format_stamp: "Formatted at 2026-09-28 22:03:09 on dist-test-slave-3t36"
server_key: "81e851203492c34c0d2e1d72fd5baa97"
server_key_iv: "8e774b22134870b4cd07986a09864303"
server_key_version: "encryptionkey@0"
I20260928 22:03:10.004714 23473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:10.005700 23488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:10.006052 23473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:10.006179 23473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "d87dbd51f4d2414bbcc7a22bf6252c84"
format_stamp: "Formatted at 2026-09-28 22:03:09 on dist-test-slave-3t36"
server_key: "81e851203492c34c0d2e1d72fd5baa97"
server_key_iv: "8e774b22134870b4cd07986a09864303"
server_key_version: "encryptionkey@0"
I20260928 22:03:10.006265 23473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:10.009944 23473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:10.012465 23473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:10.012660 23473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:10.017271 23473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41479
I20260928 22:03:10.017321 23540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41479 every 8 connection(s)
I20260928 22:03:10.017712 23473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:10.018880 23541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:10.021528 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84: Bootstrap starting.
I20260928 22:03:10.022179 23541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:10.022483 23541 log.cc:824] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:10.023365 23541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84: No bootstrap required, opened a new log
I20260928 22:03:10.025635 23541 raft_consensus.cc:348] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41479 } }
I20260928 22:03:10.025827 23541 raft_consensus.cc:374] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:10.025843 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23473
I20260928 22:03:10.025878 23541 raft_consensus.cc:729] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87dbd51f4d2414bbcc7a22bf6252c84, State: Initialized, Role: FOLLOWER
I20260928 22:03:10.025995 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:10.026077 23541 consensus_queue.cc:260] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41479 } }
I20260928 22:03:10.026188 23541 raft_consensus.cc:388] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:10.026218 2709 external_mini_cluster.cc:1428] Setting key abc27b0a1eb8e96627043758d77180bd
I20260928 22:03:10.026237 23541 raft_consensus.cc:482] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:10.026286 23541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:10.027609 23541 raft_consensus.cc:504] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41479 } }
I20260928 22:03:10.027752 23541 leader_election.cc:302] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d87dbd51f4d2414bbcc7a22bf6252c84; no voters:
I20260928 22:03:10.027962 23541 leader_election.cc:288] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:10.028031 23546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:10.028198 23546 raft_consensus.cc:686] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [term 1 LEADER]: Becoming Leader. State: Replica: d87dbd51f4d2414bbcc7a22bf6252c84, State: Running, Role: LEADER
I20260928 22:03:10.028251 23541 sys_catalog.cc:564] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:10.028337 23546 consensus_queue.cc:237] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41479 } }
I20260928 22:03:10.028867 23547 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41479 } } }
I20260928 22:03:10.029007 23547 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:10.029115 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:10.026813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40323 (local address 127.0.0.1:41479)
0928 22:03:10.027108 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:10.027133 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:10.027186 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:03:10.027340 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:10.027347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:10.027379 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:10.027544 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:10.028465 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:10.028469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:10.028470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:10.028475 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:10.028528 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:10.028543 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:10.028613 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:10.028854 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 22:03:10.028907 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":148,"threads_started":1}
I20260928 22:03:10.028853 23546 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d87dbd51f4d2414bbcc7a22bf6252c84. Latest consensus state: current_term: 1 leader_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87dbd51f4d2414bbcc7a22bf6252c84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41479 } } }
I20260928 22:03:10.029482 23546 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:10.029806 23555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:10.030350 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:10.030375 23555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:10.032435 23555 catalog_manager.cc:1347] Generated new cluster ID: 8bb3b3f0303a414fb9794e64f423a062
I20260928 22:03:10.032495 23555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:10.033248 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:10.031853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47685 (local address 127.0.0.1:41479)
0928 22:03:10.031896 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:10.031898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:10.032055 (+ 157us) server_negotiation.cc:407] Connection header received
0928 22:03:10.032091 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:10.032092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:10.032098 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:10.032199 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:10.033009 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:10.033011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:10.033011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:10.033015 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:10.033031 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:10.033036 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:10.033104 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:10.033160 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 22:03:10.033177 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:03:10.053527 23555 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:10.053793 23555 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:10.063642 23555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d87dbd51f4d2414bbcc7a22bf6252c84: Generated new TSK 0
I20260928 22:03:10.063912 23555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:10.071290 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23473
2026-09-28T22:03:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:10 dist-test-slave-3t36 krb5kdc[23572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:10 dist-test-slave-3t36 krb5kdc[23572](info): set up 2 sockets
Sep 28 22:03:10 dist-test-slave-3t36 krb5kdc[23572](info): commencing operation
krb5kdc: starting...
W20260928 22:03:11.124938 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:11 dist-test-slave-3t36 krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:11.252481 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:11.335150 23585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:11.335376 23585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:11.335400 23585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:11.336627 23585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:11.336735 23585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:11.336759 23585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:11.336778 23585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:11.336799 23585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:11.338330 23585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:11.338595 23585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:11.338850 23585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:11.344102 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:11.344247 23585 server_base.cc:1034] running on GCE node
W20260928 22:03:11.344095 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:11.344095 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:11.344647 23585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:11.345057 23585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:11.346225 23585 hybrid_clock.cc:648] HybridClock initialized: now 1790632991346204 us; error 41 us; skew 500 ppm
Sep 28 22:03:11 dist-test-slave-3t36 krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:11.349165 23585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:11.349655 23585 webserver.cc:460] Webserver started at http://127.0.0.1:35521/ using document root <none> and password file <none>
I20260928 22:03:11.349901 23585 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:11.349970 23585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:11.350150 23585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:11.351286 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "d39593d77ca740c1b7e48d7bb0adb5c6"
format_stamp: "Formatted at 2026-09-28 22:03:11 on dist-test-slave-3t36"
server_key: "f888a9014757884571b9ff4d34ffc8eb"
server_key_iv: "b5f2b515e67ca5fd3419265bec832fcb"
server_key_version: "encryptionkey@0"
I20260928 22:03:11.351660 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "d39593d77ca740c1b7e48d7bb0adb5c6"
format_stamp: "Formatted at 2026-09-28 22:03:11 on dist-test-slave-3t36"
server_key: "f888a9014757884571b9ff4d34ffc8eb"
server_key_iv: "b5f2b515e67ca5fd3419265bec832fcb"
server_key_version: "encryptionkey@0"
I20260928 22:03:11.353142 23585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:11.353993 23600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:11.354358 23585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:11.354491 23585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "d39593d77ca740c1b7e48d7bb0adb5c6"
format_stamp: "Formatted at 2026-09-28 22:03:11 on dist-test-slave-3t36"
server_key: "f888a9014757884571b9ff4d34ffc8eb"
server_key_iv: "b5f2b515e67ca5fd3419265bec832fcb"
server_key_version: "encryptionkey@0"
I20260928 22:03:11.354574 23585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:11.383706 23585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:11.385980 23585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:11.386163 23585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:11.390297 23585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39509
I20260928 22:03:11.390339 23652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39509 every 8 connection(s)
I20260928 22:03:11.390704 23585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:11.391706 23653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:11.394068 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6: Bootstrap starting.
I20260928 22:03:11.394618 23653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:11.394868 23653 log.cc:824] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:11.395614 23653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6: No bootstrap required, opened a new log
I20260928 22:03:11.397413 23653 raft_consensus.cc:348] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260928 22:03:11.397562 23653 raft_consensus.cc:374] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:11.397584 23653 raft_consensus.cc:729] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39593d77ca740c1b7e48d7bb0adb5c6, State: Initialized, Role: FOLLOWER
I20260928 22:03:11.397742 23653 consensus_queue.cc:260] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260928 22:03:11.397812 23653 raft_consensus.cc:388] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:11.397854 23653 raft_consensus.cc:482] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:11.397828 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23585
I20260928 22:03:11.397909 23653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:11.397994 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:11.398214 2709 external_mini_cluster.cc:1428] Setting key d2a2832b6d7da26f5b93d5671ed5e2c1
I20260928 22:03:11.398579 23653 raft_consensus.cc:504] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260928 22:03:11.398689 23653 leader_election.cc:302] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d39593d77ca740c1b7e48d7bb0adb5c6; no voters:
I20260928 22:03:11.398861 23653 leader_election.cc:288] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:11.398918 23657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:11.399144 23653 sys_catalog.cc:564] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:11.399161 23657 raft_consensus.cc:686] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [term 1 LEADER]: Becoming Leader. State: Replica: d39593d77ca740c1b7e48d7bb0adb5c6, State: Running, Role: LEADER
I20260928 22:03:11.399395 23657 consensus_queue.cc:237] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260928 22:03:11.399818 23659 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260928 22:03:11.399910 23659 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:11.400152 23660 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39593d77ca740c1b7e48d7bb0adb5c6. Latest consensus state: current_term: 1 leader_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39593d77ca740c1b7e48d7bb0adb5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260928 22:03:11.400547 23660 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:11.400755 23665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:11.401383 23665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:11.403172 23665 catalog_manager.cc:1347] Generated new cluster ID: 8df2a29acaef477890ec6ba90f4fa6fb
I20260928 22:03:11.403237 23665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:03:11 dist-test-slave-3t36 krb5kdc[23572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:03:11.412048 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:11.398757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:53271 (local address 127.0.0.1:39509)
0928 22:03:11.399023 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:11.399033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:11.399123 (+ 90us) server_negotiation.cc:407] Connection header received
0928 22:03:11.399369 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:11.399378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:11.399448 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:11.399573 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:11.400821 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:11.402313 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:11.403392 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:11.404026 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:11.404149 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:11.407196 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:11.407202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:11.407202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:11.407217 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:11.410133 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:11.410646 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:11.410650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:11.410651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:11.410786 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:11.411034 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:11.411035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:11.411035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:11.411368 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:11.411572 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:11.411664 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:03:11.411780 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":123,"threads_started":1}
I20260928 22:03:11.413129 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:11.424649 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:11.414670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:48181 (local address 127.0.0.1:39509)
0928 22:03:11.414710 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:11.414712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:11.416884 (+ 2172us) server_negotiation.cc:407] Connection header received
0928 22:03:11.416943 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:11.416944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:11.416957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:11.417020 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:11.417693 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:11.418490 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:11.419593 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:11.419940 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:11.420014 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:11.421598 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:11.421600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:11.421600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:11.421607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:11.423343 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:11.423794 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:11.423796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:11.423796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:11.423859 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:11.424099 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:11.424100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:11.424100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:11.424233 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:11.424340 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:11.424484 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 22:03:11.424547 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:03:11.433162 23665 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:11.433763 23665 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:11.436555 23665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d39593d77ca740c1b7e48d7bb0adb5c6: Generated new TSK 0
I20260928 22:03:11.436802 23665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:11.440237 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23585
W20260928 22:03:11.444854 23570 connection.cc:535] client connection to 127.0.0.1:39509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39509): Connection reset by peer (error 104)
2026-09-28T22:03:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:11 dist-test-slave-3t36 krb5kdc[23684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:11 dist-test-slave-3t36 krb5kdc[23684](info): set up 2 sockets
Sep 28 22:03:11 dist-test-slave-3t36 krb5kdc[23684](info): commencing operation
krb5kdc: starting...
W20260928 22:03:12.488919 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:12 dist-test-slave-3t36 krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:12.616326 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:12.701759 23697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:12.702011 23697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:12.702052 23697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:12.703322 23697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:12.703399 23697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:12.703428 23697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:12.703460 23697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:12.703485 23697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:12.705096 23697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:12.705380 23697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:12.705623 23697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:12.710734 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:12.710858 23702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:12.710830 23703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:12.710960 23697 server_base.cc:1034] running on GCE node
I20260928 22:03:12.711208 23697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:12.711513 23697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:12.712785 23697 hybrid_clock.cc:648] HybridClock initialized: now 1790632992712748 us; error 61 us; skew 500 ppm
Sep 28 22:03:12 dist-test-slave-3t36 krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:12.715464 23697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:12.715927 23697 webserver.cc:460] Webserver started at http://127.0.0.1:37721/ using document root <none> and password file <none>
I20260928 22:03:12.716123 23697 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:12.716163 23697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:12.716261 23697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:12.717372 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "48bcba42a87f48d5a1f089abb9240523"
format_stamp: "Formatted at 2026-09-28 22:03:12 on dist-test-slave-3t36"
server_key: "2b9230b4c93d4cb9616a9bc003e78e16"
server_key_iv: "0350765dbb28527fff360c2d2ca36b08"
server_key_version: "encryptionkey@0"
I20260928 22:03:12.717682 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "48bcba42a87f48d5a1f089abb9240523"
format_stamp: "Formatted at 2026-09-28 22:03:12 on dist-test-slave-3t36"
server_key: "2b9230b4c93d4cb9616a9bc003e78e16"
server_key_iv: "0350765dbb28527fff360c2d2ca36b08"
server_key_version: "encryptionkey@0"
I20260928 22:03:12.719108 23697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:03:12.719930 23712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:12.720111 23697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:12.720247 23697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "48bcba42a87f48d5a1f089abb9240523"
format_stamp: "Formatted at 2026-09-28 22:03:12 on dist-test-slave-3t36"
server_key: "2b9230b4c93d4cb9616a9bc003e78e16"
server_key_iv: "0350765dbb28527fff360c2d2ca36b08"
server_key_version: "encryptionkey@0"
I20260928 22:03:12.720328 23697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:12.735047 23697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:12.737603 23697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:12.737783 23697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:12.742244 23697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38169
I20260928 22:03:12.742283 23764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38169 every 8 connection(s)
I20260928 22:03:12.742662 23697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:12.743655 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:12.746229 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523: Bootstrap starting.
I20260928 22:03:12.746830 23765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:12.747119 23765 log.cc:824] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:12.747830 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523: No bootstrap required, opened a new log
I20260928 22:03:12.749456 23765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bcba42a87f48d5a1f089abb9240523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260928 22:03:12.749609 23765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:12.749632 23765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48bcba42a87f48d5a1f089abb9240523, State: Initialized, Role: FOLLOWER
I20260928 22:03:12.749795 23765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bcba42a87f48d5a1f089abb9240523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260928 22:03:12.749861 23765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:12.749883 23765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:12.749953 23765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:12.750515 23765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bcba42a87f48d5a1f089abb9240523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260928 22:03:12.750603 23765 leader_election.cc:302] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48bcba42a87f48d5a1f089abb9240523; no voters:
I20260928 22:03:12.750802 23765 leader_election.cc:288] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:12.750872 23768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:12.751049 23768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [term 1 LEADER]: Becoming Leader. State: Replica: 48bcba42a87f48d5a1f089abb9240523, State: Running, Role: LEADER
I20260928 22:03:12.751073 23765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:12.751181 23768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bcba42a87f48d5a1f089abb9240523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260928 22:03:12.751612 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23697
I20260928 22:03:12.751741 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:12.751762 23769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48bcba42a87f48d5a1f089abb9240523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bcba42a87f48d5a1f089abb9240523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260928 22:03:12.751842 23769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:12.751972 2709 external_mini_cluster.cc:1428] Setting key 01b81a9ee31766934b40b1ea29cda43c
I20260928 22:03:12.752197 23770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48bcba42a87f48d5a1f089abb9240523. Latest consensus state: current_term: 1 leader_uuid: "48bcba42a87f48d5a1f089abb9240523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bcba42a87f48d5a1f089abb9240523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260928 22:03:12.752378 23770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:12.752532 23777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:12.753337 23777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:12.754911 23777 catalog_manager.cc:1347] Generated new cluster ID: ddc82b16cb0c4004b5e472c7a9f2dc14
I20260928 22:03:12.754962 23777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:12.758687 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:12.752460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:44291 (local address 127.0.0.1:38169)
0928 22:03:12.753053 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:12.753063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:12.753095 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:03:12.753240 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:12.753248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:12.753319 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:12.753464 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:12.753737 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:12.755168 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:12.756319 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:12.757043 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:12.757163 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:12.757943 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:12.757948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:12.757949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:12.758170 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:12.758214 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:12.758231 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:12.758322 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:12.758419 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 22:03:12.758449 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":73,"threads_started":1}
I20260928 22:03:12.759747 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:12.760845 23777 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:12.761642 23777 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:12.765219 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:12.761429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:55365 (local address 127.0.0.1:38169)
0928 22:03:12.761467 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:12.761469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:12.761615 (+ 146us) server_negotiation.cc:407] Connection header received
0928 22:03:12.761660 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:12.761662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:12.761675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:12.761727 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:12.761921 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:12.762696 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:12.763756 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:12.764067 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:12.764102 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:12.764966 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:12.764968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:12.764968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:12.764973 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:12.764988 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:12.764992 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:12.765020 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:12.765130 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 22:03:12.765143 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:03:12.767854 23777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48bcba42a87f48d5a1f089abb9240523: Generated new TSK 0
I20260928 22:03:12.768074 23777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:12.771245 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23697
W20260928 22:03:12.775520 23682 connection.cc:535] client connection to 127.0.0.1:38169 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38169): Connection reset by peer (error 104)
2026-09-28T22:03:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:12 dist-test-slave-3t36 krb5kdc[23796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:12 dist-test-slave-3t36 krb5kdc[23796](info): set up 2 sockets
Sep 28 22:03:12 dist-test-slave-3t36 krb5kdc[23796](info): commencing operation
krb5kdc: starting...
W20260928 22:03:13.825043 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:13 dist-test-slave-3t36 krb5kdc[23796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:13.957522 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:14.040347 23809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:14.040634 23809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:14.040676 23809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:14.041962 23809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:14.042052 23809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:14.042075 23809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:14.042109 23809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:14.042157 23809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:14.043779 23809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:14.044086 23809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:14.044335 23809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:14.049674 23815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:14.049865 23817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:14.049758 23814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:14.049698 23809 server_base.cc:1034] running on GCE node
I20260928 22:03:14.050251 23809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:14.050561 23809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:14.051736 23809 hybrid_clock.cc:648] HybridClock initialized: now 1790632994051708 us; error 40 us; skew 500 ppm
Sep 28 22:03:14 dist-test-slave-3t36 krb5kdc[23796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:14.059744 23809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:14.060307 23809 webserver.cc:460] Webserver started at http://127.0.0.1:40113/ using document root <none> and password file <none>
I20260928 22:03:14.060595 23809 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:14.060653 23809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:14.060812 23809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:14.061969 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "892db9b18efe4f0ea2649dff705d0008"
format_stamp: "Formatted at 2026-09-28 22:03:14 on dist-test-slave-3t36"
server_key: "7964873b9eefe5943e8c58b67c94e748"
server_key_iv: "8f62a7e5db817e24660cc6e630b1b3ee"
server_key_version: "encryptionkey@0"
I20260928 22:03:14.062353 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "892db9b18efe4f0ea2649dff705d0008"
format_stamp: "Formatted at 2026-09-28 22:03:14 on dist-test-slave-3t36"
server_key: "7964873b9eefe5943e8c58b67c94e748"
server_key_iv: "8f62a7e5db817e24660cc6e630b1b3ee"
server_key_version: "encryptionkey@0"
I20260928 22:03:14.063900 23809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:03:14.064901 23824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:14.065181 23809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:14.065325 23809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "892db9b18efe4f0ea2649dff705d0008"
format_stamp: "Formatted at 2026-09-28 22:03:14 on dist-test-slave-3t36"
server_key: "7964873b9eefe5943e8c58b67c94e748"
server_key_iv: "8f62a7e5db817e24660cc6e630b1b3ee"
server_key_version: "encryptionkey@0"
I20260928 22:03:14.065418 23809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:14.069070 23809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:14.072319 23809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:14.072450 23809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:14.077042 23809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45873
I20260928 22:03:14.077122 23876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45873 every 8 connection(s)
I20260928 22:03:14.077454 23809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:14.078482 23877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:14.080946 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008: Bootstrap starting.
I20260928 22:03:14.081550 23877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:14.081842 23877 log.cc:824] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:14.082603 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008: No bootstrap required, opened a new log
I20260928 22:03:14.083041 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23809
I20260928 22:03:14.083166 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:14.083365 2709 external_mini_cluster.cc:1428] Setting key 534ead11b4c5cfbe14a6729c56becd62
I20260928 22:03:14.084743 23877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892db9b18efe4f0ea2649dff705d0008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45873 } }
I20260928 22:03:14.084932 23877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:14.085031 23877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892db9b18efe4f0ea2649dff705d0008, State: Initialized, Role: FOLLOWER
I20260928 22:03:14.085311 23877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892db9b18efe4f0ea2649dff705d0008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45873 } }
I20260928 22:03:14.085412 23877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:14.085438 23877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:14.085512 23877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:14.086207 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:14.083936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:58925 (local address 127.0.0.1:45873)
0928 22:03:14.084144 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:14.084152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:14.084180 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:03:14.084728 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:14.084736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:14.084768 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:14.084898 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:14.085725 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:14.085728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:14.085728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:14.085732 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:14.085772 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:14.085783 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:14.085839 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:14.085929 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 22:03:14.085988 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260928 22:03:14.086728 23877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892db9b18efe4f0ea2649dff705d0008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45873 } }
I20260928 22:03:14.086869 23877 leader_election.cc:302] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 892db9b18efe4f0ea2649dff705d0008; no voters:
I20260928 22:03:14.087059 23877 leader_election.cc:288] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:14.087116 23882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:14.087288 23882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [term 1 LEADER]: Becoming Leader. State: Replica: 892db9b18efe4f0ea2649dff705d0008, State: Running, Role: LEADER
I20260928 22:03:14.087345 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:14.087337 23877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:14.087453 23882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892db9b18efe4f0ea2649dff705d0008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45873 } }
I20260928 22:03:14.087781 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892db9b18efe4f0ea2649dff705d0008. Latest consensus state: current_term: 1 leader_uuid: "892db9b18efe4f0ea2649dff705d0008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892db9b18efe4f0ea2649dff705d0008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45873 } } }
I20260928 22:03:14.087852 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:14.087989 23883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892db9b18efe4f0ea2649dff705d0008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892db9b18efe4f0ea2649dff705d0008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45873 } } }
I20260928 22:03:14.088040 23883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008 [sys.catalog]: This master's current role is: LEADER
W20260928 22:03:14.089264 23902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:14.089353 23902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:14.089416 23889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:14.089600 23889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:14.090360 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:14.089099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:55551 (local address 127.0.0.1:45873)
0928 22:03:14.089138 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:14.089141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:14.089272 (+ 131us) server_negotiation.cc:407] Connection header received
0928 22:03:14.089337 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:14.089337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:14.089344 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:14.089391 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:14.090119 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:14.090121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:14.090121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:14.090126 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:14.090143 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:14.090147 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:14.090204 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:14.090260 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 22:03:14.090276 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:03:14.091231 23889 catalog_manager.cc:1347] Generated new cluster ID: 98950ea119ef4798af15bb58287269de
I20260928 22:03:14.091284 23889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:14.113404 23889 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:14.113672 23889 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:14.121912 23889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 892db9b18efe4f0ea2649dff705d0008: Generated new TSK 0
I20260928 22:03:14.122175 23889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:14.129886 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23809
2026-09-28T22:03:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:14 dist-test-slave-3t36 krb5kdc[23908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:14 dist-test-slave-3t36 krb5kdc[23908](info): set up 2 sockets
Sep 28 22:03:14 dist-test-slave-3t36 krb5kdc[23908](info): commencing operation
krb5kdc: starting...
W20260928 22:03:15.193188 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:15 dist-test-slave-3t36 krb5kdc[23908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:15.320293 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:15.406759 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:15.407030 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:15.407071 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:15.408280 23921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:15.408357 23921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:15.408382 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:15.408403 23921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:15.408433 23921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:15.410619 23921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:15.410954 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:15.411201 23921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:15.416761 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:15.416766 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:15.416766 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:15.417022 23921 server_base.cc:1034] running on GCE node
I20260928 22:03:15.417337 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:15.417711 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:15.418928 23921 hybrid_clock.cc:648] HybridClock initialized: now 1790632995418925 us; error 48 us; skew 500 ppm
Sep 28 22:03:15 dist-test-slave-3t36 krb5kdc[23908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:15.422017 23921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:15.422559 23921 webserver.cc:460] Webserver started at http://127.0.0.1:42565/ using document root <none> and password file <none>
I20260928 22:03:15.422837 23921 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:15.422909 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:15.423062 23921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:15.424216 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "220de3a6fe5243c693899cc9e960a7eb"
format_stamp: "Formatted at 2026-09-28 22:03:15 on dist-test-slave-3t36"
server_key: "79dacff3e785d838c5b64004007b79b3"
server_key_iv: "04608eb4eca6b119f0427d4a4d4b1fcd"
server_key_version: "encryptionkey@0"
I20260928 22:03:15.424587 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "220de3a6fe5243c693899cc9e960a7eb"
format_stamp: "Formatted at 2026-09-28 22:03:15 on dist-test-slave-3t36"
server_key: "79dacff3e785d838c5b64004007b79b3"
server_key_iv: "04608eb4eca6b119f0427d4a4d4b1fcd"
server_key_version: "encryptionkey@0"
I20260928 22:03:15.426072 23921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:15.426891 23936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:15.427073 23921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:15.427187 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "220de3a6fe5243c693899cc9e960a7eb"
format_stamp: "Formatted at 2026-09-28 22:03:15 on dist-test-slave-3t36"
server_key: "79dacff3e785d838c5b64004007b79b3"
server_key_iv: "04608eb4eca6b119f0427d4a4d4b1fcd"
server_key_version: "encryptionkey@0"
I20260928 22:03:15.427276 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:15.447067 23921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:15.449992 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:15.450232 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:15.454977 23921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39883
I20260928 22:03:15.455055 23988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39883 every 8 connection(s)
I20260928 22:03:15.455442 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:15.456271 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 23921
I20260928 22:03:15.456410 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:15.456626 2709 external_mini_cluster.cc:1428] Setting key 53f0e5d9cdaff212ef9c6a2e2a515399
I20260928 22:03:15.456764 23989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:15.460199 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb: Bootstrap starting.
I20260928 22:03:15.460973 23989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:15.461308 23989 log.cc:824] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:15.462035 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb: No bootstrap required, opened a new log
Sep 28 22:03:15 dist-test-slave-3t36 krb5kdc[23908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:03:15.464148 23989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220de3a6fe5243c693899cc9e960a7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39883 } }
I20260928 22:03:15.464291 23989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:15.464322 23989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220de3a6fe5243c693899cc9e960a7eb, State: Initialized, Role: FOLLOWER
I20260928 22:03:15.464483 23989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220de3a6fe5243c693899cc9e960a7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39883 } }
I20260928 22:03:15.464547 23989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:15.464600 23989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:15.464692 23989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:15.465478 23989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220de3a6fe5243c693899cc9e960a7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39883 } }
I20260928 22:03:15.465574 23989 leader_election.cc:302] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220de3a6fe5243c693899cc9e960a7eb; no voters:
I20260928 22:03:15.465796 23989 leader_election.cc:288] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:15.465842 23994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:15.465947 23994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [term 1 LEADER]: Becoming Leader. State: Replica: 220de3a6fe5243c693899cc9e960a7eb, State: Running, Role: LEADER
I20260928 22:03:15.466022 23994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220de3a6fe5243c693899cc9e960a7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39883 } }
I20260928 22:03:15.466071 23989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:15.466343 23994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "220de3a6fe5243c693899cc9e960a7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220de3a6fe5243c693899cc9e960a7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39883 } } }
I20260928 22:03:15.466473 23994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:15.466642 23994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 220de3a6fe5243c693899cc9e960a7eb. Latest consensus state: current_term: 1 leader_uuid: "220de3a6fe5243c693899cc9e960a7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220de3a6fe5243c693899cc9e960a7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39883 } } }
I20260928 22:03:15.466766 23994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:15.466853 23999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:15.467478 23999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:15.468887 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:15.457241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:40665 (local address 127.0.0.1:39883)
0928 22:03:15.457432 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:15.457442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:15.457478 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:03:15.457932 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:15.457939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:15.457991 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:15.458110 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:15.458966 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:15.459970 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:15.461081 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:15.461600 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:15.461744 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:15.464713 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:15.464718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:15.464718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:15.464732 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:15.466973 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:15.467514 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:15.467517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:15.467518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:15.467628 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:15.467909 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:15.467911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:15.467911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:15.468225 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:15.468411 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:15.468542 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 22:03:15.468649 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260928 22:03:15.469812 23999 catalog_manager.cc:1347] Generated new cluster ID: cc521ac6419c43b2982f146e80476d23
I20260928 22:03:15.469877 23999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:15.470068 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:15.478045 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:15.471500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:52111 (local address 127.0.0.1:39883)
0928 22:03:15.471540 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:15.471542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:15.471690 (+ 148us) server_negotiation.cc:407] Connection header received
0928 22:03:15.471760 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:15.471761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:15.471771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:15.471827 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:15.472363 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:15.472965 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:15.473719 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:15.473955 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:15.474037 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:15.475285 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:15.475287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:15.475287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:15.475294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:15.476826 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:15.477205 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:15.477207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:15.477207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:15.477264 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:15.477494 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:15.477495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:15.477496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:15.477619 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:15.477734 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:15.477856 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 22:03:15.477923 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":20}
I20260928 22:03:15.491473 23999 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:15.492259 23999 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:15.495543 23999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 220de3a6fe5243c693899cc9e960a7eb: Generated new TSK 0
I20260928 22:03:15.495760 23999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:15.503647 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 23921
W20260928 22:03:15.508209 23906 connection.cc:535] client connection to 127.0.0.1:39883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39883): Connection reset by peer (error 104)
2026-09-28T22:03:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:15 dist-test-slave-3t36 krb5kdc[24020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:15 dist-test-slave-3t36 krb5kdc[24020](info): set up 2 sockets
Sep 28 22:03:15 dist-test-slave-3t36 krb5kdc[24020](info): commencing operation
krb5kdc: starting...
W20260928 22:03:16.545022 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:16 dist-test-slave-3t36 krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:16.691337 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:16.776537 24033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:16.776788 24033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:16.776816 24033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:16.778080 24033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:16.778134 24033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:16.778149 24033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:16.778162 24033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:16.778173 24033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:16.779757 24033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:16.780042 24033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:16.780294 24033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:16.785401 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:16.785497 24033 server_base.cc:1034] running on GCE node
W20260928 22:03:16.785517 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:16.785526 24041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:16.786026 24033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:16.786335 24033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:16.787493 24033 hybrid_clock.cc:648] HybridClock initialized: now 1790632996787461 us; error 58 us; skew 500 ppm
Sep 28 22:03:16 dist-test-slave-3t36 krb5kdc[24020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:16.790105 24033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:16.790632 24033 webserver.cc:460] Webserver started at http://127.0.0.1:36155/ using document root <none> and password file <none>
I20260928 22:03:16.791002 24033 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:16.791183 24033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:16.791365 24033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:16.792517 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "7b07f0a969b74413b61ba2a56a91a733"
format_stamp: "Formatted at 2026-09-28 22:03:16 on dist-test-slave-3t36"
server_key: "2e73504302805f14d14949ce51af2d5f"
server_key_iv: "76937759000e8eddbe285c1b8a1bc5ad"
server_key_version: "encryptionkey@0"
I20260928 22:03:16.792932 24033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "7b07f0a969b74413b61ba2a56a91a733"
format_stamp: "Formatted at 2026-09-28 22:03:16 on dist-test-slave-3t36"
server_key: "2e73504302805f14d14949ce51af2d5f"
server_key_iv: "76937759000e8eddbe285c1b8a1bc5ad"
server_key_version: "encryptionkey@0"
I20260928 22:03:16.794322 24033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:16.795152 24048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:16.795367 24033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:16.795486 24033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "7b07f0a969b74413b61ba2a56a91a733"
format_stamp: "Formatted at 2026-09-28 22:03:16 on dist-test-slave-3t36"
server_key: "2e73504302805f14d14949ce51af2d5f"
server_key_iv: "76937759000e8eddbe285c1b8a1bc5ad"
server_key_version: "encryptionkey@0"
I20260928 22:03:16.795567 24033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:16.803020 24033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:16.805227 24033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:16.805404 24033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:16.809547 24033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39475
I20260928 22:03:16.809602 24100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39475 every 8 connection(s)
I20260928 22:03:16.809998 24033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:16.810920 24101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:16.813284 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733: Bootstrap starting.
I20260928 22:03:16.813874 24101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:16.814158 24101 log.cc:824] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:16.814931 24101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733: No bootstrap required, opened a new log
I20260928 22:03:16.816776 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24033
I20260928 22:03:16.816627 24101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b07f0a969b74413b61ba2a56a91a733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39475 } }
I20260928 22:03:16.816859 24101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:16.816902 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:16.816900 24101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b07f0a969b74413b61ba2a56a91a733, State: Initialized, Role: FOLLOWER
I20260928 22:03:16.817062 24101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b07f0a969b74413b61ba2a56a91a733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39475 } }
I20260928 22:03:16.817111 2709 external_mini_cluster.cc:1428] Setting key 04597a6928aa753efb6363e47b850775
I20260928 22:03:16.817147 24101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:16.817193 24101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:16.817246 24101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:16.818076 24101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b07f0a969b74413b61ba2a56a91a733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39475 } }
I20260928 22:03:16.818202 24101 leader_election.cc:302] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b07f0a969b74413b61ba2a56a91a733; no voters:
I20260928 22:03:16.818403 24101 leader_election.cc:288] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:16.818425 24106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:16.818535 24106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [term 1 LEADER]: Becoming Leader. State: Replica: 7b07f0a969b74413b61ba2a56a91a733, State: Running, Role: LEADER
I20260928 22:03:16.818676 24101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:16.818673 24106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b07f0a969b74413b61ba2a56a91a733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39475 } }
I20260928 22:03:16.819132 24108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b07f0a969b74413b61ba2a56a91a733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b07f0a969b74413b61ba2a56a91a733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39475 } } }
I20260928 22:03:16.819427 24108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:16.819344 24107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b07f0a969b74413b61ba2a56a91a733. Latest consensus state: current_term: 1 leader_uuid: "7b07f0a969b74413b61ba2a56a91a733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b07f0a969b74413b61ba2a56a91a733" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39475 } } }
I20260928 22:03:16.819872 24107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:16.820032 24115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:16.820765 24115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:16.822497 24115 catalog_manager.cc:1347] Generated new cluster ID: 12fb0c4d79b845189340d654bea8be2c
I20260928 22:03:16.822551 24115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:16.823632 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:16.817648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:36777 (local address 127.0.0.1:39475)
0928 22:03:16.818156 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:16.818167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:16.818205 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:03:16.818331 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:16.818338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:16.818399 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:16.818811 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:16.818915 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:16.820396 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:16.821370 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:16.822008 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:16.822141 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:16.823005 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:16.823009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:16.823009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:16.823195 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:16.823227 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:16.823239 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:16.823336 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:16.823419 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:03:16.823449 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":422,"threads_started":1}
I20260928 22:03:16.824599 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:16.832069 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:16.826243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:49547 (local address 127.0.0.1:39475)
0928 22:03:16.826283 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:16.826285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:16.828941 (+ 2656us) server_negotiation.cc:407] Connection header received
0928 22:03:16.829016 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:16.829018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:16.829030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:16.829104 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:16.829305 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:16.829832 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:16.830579 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:16.830914 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:16.830992 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:16.831753 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:16.831754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:16.831754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:16.831759 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:16.831776 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:16.831780 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:16.831840 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:16.831920 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 22:03:16.831939 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":20}
I20260928 22:03:16.853418 24115 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:16.854207 24115 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:16.863860 24115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b07f0a969b74413b61ba2a56a91a733: Generated new TSK 0
I20260928 22:03:16.864099 24115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:16.869801 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24033
W20260928 22:03:16.874100 24018 connection.cc:535] client connection to 127.0.0.1:39475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39475): Connection reset by peer (error 104)
2026-09-28T22:03:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:16 dist-test-slave-3t36 krb5kdc[24132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:16 dist-test-slave-3t36 krb5kdc[24132](info): set up 2 sockets
Sep 28 22:03:16 dist-test-slave-3t36 krb5kdc[24132](info): commencing operation
krb5kdc: starting...
W20260928 22:03:17.912940 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:17 dist-test-slave-3t36 krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:03:18.035965 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:18.116463 24145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:18.116729 24145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:18.116770 24145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:18.117949 24145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:18.118026 24145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:18.118048 24145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:18.118069 24145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:18.118100 24145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:18.119586 24145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:18.119874 24145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:18.120110 24145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:18.125471 24150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:18.125465 24151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:18.125519 24145 server_base.cc:1034] running on GCE node
W20260928 22:03:18.125465 24153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:18.125862 24145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:18.126109 24145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:18.127254 24145 hybrid_clock.cc:648] HybridClock initialized: now 1790632998127231 us; error 36 us; skew 500 ppm
Sep 28 22:03:18 dist-test-slave-3t36 krb5kdc[24132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:18.130057 24145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:03:18.130599 24145 webserver.cc:460] Webserver started at http://127.0.0.1:34185/ using document root <none> and password file <none>
I20260928 22:03:18.130868 24145 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:18.130944 24145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:18.131093 24145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:18.132198 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "fd19174fda544fa0ae09c4a5605872f7"
format_stamp: "Formatted at 2026-09-28 22:03:18 on dist-test-slave-3t36"
server_key: "175dcb6876a0145db3bcd2975036c526"
server_key_iv: "60069b6f1724a74baa03ef35ea9507fe"
server_key_version: "encryptionkey@0"
I20260928 22:03:18.132622 24145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "fd19174fda544fa0ae09c4a5605872f7"
format_stamp: "Formatted at 2026-09-28 22:03:18 on dist-test-slave-3t36"
server_key: "175dcb6876a0145db3bcd2975036c526"
server_key_iv: "60069b6f1724a74baa03ef35ea9507fe"
server_key_version: "encryptionkey@0"
I20260928 22:03:18.134088 24145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:18.135025 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:18.135270 24145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:18.135394 24145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "fd19174fda544fa0ae09c4a5605872f7"
format_stamp: "Formatted at 2026-09-28 22:03:18 on dist-test-slave-3t36"
server_key: "175dcb6876a0145db3bcd2975036c526"
server_key_iv: "60069b6f1724a74baa03ef35ea9507fe"
server_key_version: "encryptionkey@0"
I20260928 22:03:18.135486 24145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:18.139334 24145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:18.141688 24145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:18.141803 24145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:18.145694 24145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45911
I20260928 22:03:18.145787 24212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45911 every 8 connection(s)
I20260928 22:03:18.146103 24145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:18.147269 24213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:18.149715 24213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7: Bootstrap starting.
I20260928 22:03:18.150326 24213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:18.150631 24213 log.cc:824] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:18.150876 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24145
I20260928 22:03:18.151003 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:18.151198 2709 external_mini_cluster.cc:1428] Setting key 3d77e1425c8a3e779996f8bd7a1cef0c
I20260928 22:03:18.151502 24213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7: No bootstrap required, opened a new log
I20260928 22:03:18.153430 24213 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19174fda544fa0ae09c4a5605872f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260928 22:03:18.153656 24213 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:18.153751 24213 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd19174fda544fa0ae09c4a5605872f7, State: Initialized, Role: FOLLOWER
I20260928 22:03:18.153951 24213 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19174fda544fa0ae09c4a5605872f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260928 22:03:18.154069 24213 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:18.154114 24213 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:18.154181 24213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:18.154871 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:18.151776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:58797 (local address 127.0.0.1:45911)
0928 22:03:18.151965 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:18.151974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:18.152001 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:03:18.153410 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:18.153418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:18.153449 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:18.153579 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:18.154394 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:18.154397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:18.154397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:18.154401 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:18.154440 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:18.154452 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:18.154513 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:18.154610 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 22:03:18.154667 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260928 22:03:18.155352 24213 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19174fda544fa0ae09c4a5605872f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260928 22:03:18.155480 24213 leader_election.cc:302] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd19174fda544fa0ae09c4a5605872f7; no voters:
I20260928 22:03:18.155695 24213 leader_election.cc:288] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:18.155725 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:18.155741 24218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:18.155879 24218 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [term 1 LEADER]: Becoming Leader. State: Replica: fd19174fda544fa0ae09c4a5605872f7, State: Running, Role: LEADER
I20260928 22:03:18.155968 24213 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:18.155956 24218 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19174fda544fa0ae09c4a5605872f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260928 22:03:18.156338 24220 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd19174fda544fa0ae09c4a5605872f7. Latest consensus state: current_term: 1 leader_uuid: "fd19174fda544fa0ae09c4a5605872f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19174fda544fa0ae09c4a5605872f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260928 22:03:18.156342 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd19174fda544fa0ae09c4a5605872f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd19174fda544fa0ae09c4a5605872f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260928 22:03:18.156436 24220 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:18.156445 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:18.157333 24226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:18.158111 24226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:18.158955 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:18.157534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:49743 (local address 127.0.0.1:45911)
0928 22:03:18.157594 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:18.157596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:18.157939 (+ 343us) server_negotiation.cc:407] Connection header received
0928 22:03:18.158016 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:18.158017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:18.158022 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:18.158079 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:18.158726 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:18.158727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:18.158728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:18.158732 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:18.158748 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:18.158753 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:18.158800 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:18.158857 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 22:03:18.158872 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:03:18.160054 24226 catalog_manager.cc:1347] Generated new cluster ID: d3cc32c6dd3e4f5d8fd9adc1533dd040
I20260928 22:03:18.160118 24226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:18.173794 24226 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:18.174055 24226 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:18.177168 24226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd19174fda544fa0ae09c4a5605872f7: Generated new TSK 0
I20260928 22:03:18.177394 24226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:18.182952 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24145
2026-09-28T22:03:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:18 dist-test-slave-3t36 krb5kdc[24244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:18 dist-test-slave-3t36 krb5kdc[24244](info): set up 2 sockets
Sep 28 22:03:18 dist-test-slave-3t36 krb5kdc[24244](info): commencing operation
krb5kdc: starting...
W20260928 22:03:19.252903 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:19 dist-test-slave-3t36 krb5kdc[24244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:19.375599 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:44287
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:19.456570 24257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:19.456830 24257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:19.456853 24257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:19.458102 24257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:19.458164 24257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:19.458178 24257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:19.458192 24257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:19.458204 24257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:19.459781 24257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:44287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:44287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:19.460050 24257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:19.460307 24257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:19.465550 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:19.465555 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:19.465555 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:19.465704 24257 server_base.cc:1034] running on GCE node
I20260928 22:03:19.466039 24257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:19.466375 24257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:19.467556 24257 hybrid_clock.cc:648] HybridClock initialized: now 1790632999467553 us; error 44 us; skew 500 ppm
Sep 28 22:03:19 dist-test-slave-3t36 krb5kdc[24244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:19.476545 24257 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:19.477190 24257 webserver.cc:460] Webserver started at http://127.2.165.126:36927/ using document root <none> and password file <none>
I20260928 22:03:19.477468 24257 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:19.477551 24257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:19.477720 24257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:19.478950 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "d5f0ae0749eb43aebda5b77b65a01049"
format_stamp: "Formatted at 2026-09-28 22:03:19 on dist-test-slave-3t36"
server_key: "1703bbd9a5b9bdf5acdbdf4deb8ba942"
server_key_iv: "1baec9a94c2229dee3ebbe741543ba5d"
server_key_version: "encryptionkey@0"
I20260928 22:03:19.479337 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "d5f0ae0749eb43aebda5b77b65a01049"
format_stamp: "Formatted at 2026-09-28 22:03:19 on dist-test-slave-3t36"
server_key: "1703bbd9a5b9bdf5acdbdf4deb8ba942"
server_key_iv: "1baec9a94c2229dee3ebbe741543ba5d"
server_key_version: "encryptionkey@0"
I20260928 22:03:19.480836 24257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:19.481652 24272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:19.481881 24257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:19.481992 24257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "d5f0ae0749eb43aebda5b77b65a01049"
format_stamp: "Formatted at 2026-09-28 22:03:19 on dist-test-slave-3t36"
server_key: "1703bbd9a5b9bdf5acdbdf4deb8ba942"
server_key_iv: "1baec9a94c2229dee3ebbe741543ba5d"
server_key_version: "encryptionkey@0"
I20260928 22:03:19.482081 24257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:19.499557 24257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:19.502087 24257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:19.502278 24257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:19.506999 24257 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:44287
I20260928 22:03:19.507069 24324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:44287 every 8 connection(s)
I20260928 22:03:19.507455 24257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:19.508399 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:19.511022 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24257
I20260928 22:03:19.511023 24325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049: Bootstrap starting.
I20260928 22:03:19.511157 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:19.511339 2709 external_mini_cluster.cc:1428] Setting key 3d2991f38f9397df86f1f567c1a18368
I20260928 22:03:19.511847 24325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:19.512303 24325 log.cc:824] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:19.513209 24325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049: No bootstrap required, opened a new log
I20260928 22:03:19.515647 24325 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f0ae0749eb43aebda5b77b65a01049" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44287 } }
I20260928 22:03:19.515815 24325 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:19.515838 24325 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f0ae0749eb43aebda5b77b65a01049, State: Initialized, Role: FOLLOWER
I20260928 22:03:19.515975 24325 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f0ae0749eb43aebda5b77b65a01049" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44287 } }
I20260928 22:03:19.516052 24325 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:19.516104 24325 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:19.516155 24325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:19.516906 24325 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f0ae0749eb43aebda5b77b65a01049" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44287 } }
I20260928 22:03:19.517019 24325 leader_election.cc:302] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f0ae0749eb43aebda5b77b65a01049; no voters:
I20260928 22:03:19.517200 24325 leader_election.cc:288] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:19.517233 24330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:19.517465 24330 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [term 1 LEADER]: Becoming Leader. State: Replica: d5f0ae0749eb43aebda5b77b65a01049, State: Running, Role: LEADER
I20260928 22:03:19.517616 24330 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f0ae0749eb43aebda5b77b65a01049" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44287 } }
I20260928 22:03:19.517491 24325 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:19.518059 24331 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f0ae0749eb43aebda5b77b65a01049. Latest consensus state: current_term: 1 leader_uuid: "d5f0ae0749eb43aebda5b77b65a01049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f0ae0749eb43aebda5b77b65a01049" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44287 } } }
I20260928 22:03:19.518210 24331 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:19.518648 24332 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f0ae0749eb43aebda5b77b65a01049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f0ae0749eb43aebda5b77b65a01049" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 44287 } } }
I20260928 22:03:19.518733 24332 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:19.518939 24334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:19.519476 24334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:03:19 dist-test-slave-3t36 krb5kdc[24244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:19.521400 24334 catalog_manager.cc:1347] Generated new cluster ID: 1a470f7c2a1a4f778e832cb82d4a7d84
I20260928 22:03:19.521780 24334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:19.524941 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:19.511947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48267 (local address 127.2.165.126:44287)
0928 22:03:19.512172 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:19.512194 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:19.512261 (+ 67us) server_negotiation.cc:407] Connection header received
0928 22:03:19.512984 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:19.512994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:19.513061 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:19.513229 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:19.513957 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:19.515485 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:19.516540 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:19.517233 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:19.517716 (+ 483us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:19.520813 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:19.520817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:19.520817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:19.520837 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:19.522597 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:19.523158 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:19.523161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:19.523162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:19.523295 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:19.523579 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:19.523579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:19.523580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:19.523910 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:19.524463 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:19.524566 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 22:03:19.524709 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
I20260928 22:03:19.526469 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:19.535732 24334 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:19.535902 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:19.528084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41781 (local address 127.2.165.126:44287)
0928 22:03:19.528125 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:19.528127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:19.528135 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:19.528159 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:19.528159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:19.528169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:19.528237 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:19.528864 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:19.529652 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:19.530659 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:19.530989 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:19.531067 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:19.532561 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:19.532563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:19.532563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:19.532572 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:19.534407 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:19.534847 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:19.534848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:19.534849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:19.534914 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:19.535176 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:19.535177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:19.535178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:19.535353 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:19.535439 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:19.535722 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 22:03:19.535795 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:03:19.536640 24334 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:19.543113 24334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f0ae0749eb43aebda5b77b65a01049: Generated new TSK 0
I20260928 22:03:19.543315 24334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:19.546959 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24257
W20260928 22:03:19.551579 24242 connection.cc:535] client connection to 127.2.165.126:44287 recv error: Network error: failed to read from TLS socket (remote: 127.2.165.126:44287): Connection reset by peer (error 104)
2026-09-28T22:03:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:19 dist-test-slave-3t36 krb5kdc[24356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:19 dist-test-slave-3t36 krb5kdc[24356](info): set up 2 sockets
Sep 28 22:03:19 dist-test-slave-3t36 krb5kdc[24356](info): commencing operation
krb5kdc: starting...
W20260928 22:03:20.596928 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:20 dist-test-slave-3t36 krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:20.720433 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:37661
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:37661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:20.805094 24369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:20.805389 24369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:20.805436 24369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:20.806695 24369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:20.806777 24369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:20.806802 24369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:20.806824 24369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:20.806855 24369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:20.808440 24369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:37661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:37661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:20.808781 24369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:20.809039 24369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:20.814499 24377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:20.814486 24374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:20.814486 24375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:20.814514 24369 server_base.cc:1034] running on GCE node
I20260928 22:03:20.814985 24369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:20.815320 24369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:20.816493 24369 hybrid_clock.cc:648] HybridClock initialized: now 1790633000816482 us; error 41 us; skew 500 ppm
Sep 28 22:03:20 dist-test-slave-3t36 krb5kdc[24356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:20.819264 24369 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:20.819726 24369 webserver.cc:460] Webserver started at http://127.2.165.126:41269/ using document root <none> and password file <none>
I20260928 22:03:20.819977 24369 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:20.820047 24369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:20.820201 24369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:20.821416 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "ef955ac30a8b44a7a6609568bea4be73"
format_stamp: "Formatted at 2026-09-28 22:03:20 on dist-test-slave-3t36"
server_key: "0fbd7525eddc2ee77c25e3b14d57ac59"
server_key_iv: "31d02ca3802615092eb06ec192118b52"
server_key_version: "encryptionkey@0"
I20260928 22:03:20.821965 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "ef955ac30a8b44a7a6609568bea4be73"
format_stamp: "Formatted at 2026-09-28 22:03:20 on dist-test-slave-3t36"
server_key: "0fbd7525eddc2ee77c25e3b14d57ac59"
server_key_iv: "31d02ca3802615092eb06ec192118b52"
server_key_version: "encryptionkey@0"
I20260928 22:03:20.823514 24369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:03:20.824465 24384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:20.824806 24369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:20.824931 24369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "ef955ac30a8b44a7a6609568bea4be73"
format_stamp: "Formatted at 2026-09-28 22:03:20 on dist-test-slave-3t36"
server_key: "0fbd7525eddc2ee77c25e3b14d57ac59"
server_key_iv: "31d02ca3802615092eb06ec192118b52"
server_key_version: "encryptionkey@0"
I20260928 22:03:20.825029 24369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:20.837141 24369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:20.839856 24369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:20.840065 24369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:20.844185 24369 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:37661
I20260928 22:03:20.844273 24436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:37661 every 8 connection(s)
I20260928 22:03:20.844661 24369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:20.845715 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24369
I20260928 22:03:20.845858 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:20.845913 24437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:20.846079 2709 external_mini_cluster.cc:1428] Setting key 25975f0fc7f604cd560fc99b677d8673
I20260928 22:03:20.850010 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73: Bootstrap starting.
I20260928 22:03:20.850693 24437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:20.851022 24437 log.cc:824] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:20.851897 24437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73: No bootstrap required, opened a new log
I20260928 22:03:20.853165 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:20.846702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50181 (local address 127.2.165.126:37661)
0928 22:03:20.846918 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:20.846928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:20.846965 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:03:20.847917 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:20.847929 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:20.848004 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:20.848226 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:20.848492 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:20.849904 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:20.850821 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:20.851429 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:20.851566 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:20.852412 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:20.852417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:20.852417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:20.852595 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:20.852627 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:20.852639 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:20.852773 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:20.852891 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 22:03:20.852933 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260928 22:03:20.853670 24437 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef955ac30a8b44a7a6609568bea4be73" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37661 } }
I20260928 22:03:20.853858 24437 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:20.853932 24437 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef955ac30a8b44a7a6609568bea4be73, State: Initialized, Role: FOLLOWER
I20260928 22:03:20.854092 24437 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef955ac30a8b44a7a6609568bea4be73" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37661 } }
I20260928 22:03:20.854182 24437 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:20.854226 24437 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:20.854274 24437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:20.854586 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:20.855079 24437 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef955ac30a8b44a7a6609568bea4be73" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37661 } }
I20260928 22:03:20.855211 24437 leader_election.cc:302] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef955ac30a8b44a7a6609568bea4be73; no voters:
I20260928 22:03:20.855414 24437 leader_election.cc:288] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:20.855538 24442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:20.855803 24442 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [term 1 LEADER]: Becoming Leader. State: Replica: ef955ac30a8b44a7a6609568bea4be73, State: Running, Role: LEADER
I20260928 22:03:20.855826 24437 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:20.855967 24442 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef955ac30a8b44a7a6609568bea4be73" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37661 } }
I20260928 22:03:20.856408 24446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef955ac30a8b44a7a6609568bea4be73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef955ac30a8b44a7a6609568bea4be73" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37661 } } }
I20260928 22:03:20.856521 24446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:20.856395 24443 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef955ac30a8b44a7a6609568bea4be73. Latest consensus state: current_term: 1 leader_uuid: "ef955ac30a8b44a7a6609568bea4be73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef955ac30a8b44a7a6609568bea4be73" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 37661 } } }
I20260928 22:03:20.856849 24443 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:20.857328 24455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:20.857859 24455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:20.860020 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:20.856261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59295 (local address 127.2.165.126:37661)
0928 22:03:20.856305 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:20.856307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:20.856464 (+ 157us) server_negotiation.cc:407] Connection header received
0928 22:03:20.856595 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:20.856596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:20.856606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:20.856639 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:20.856902 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:20.857747 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:20.858552 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:20.858870 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:20.858938 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:20.859787 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:20.859789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:20.859789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:20.859795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:20.859811 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:20.859815 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:20.859870 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:20.859932 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 22:03:20.859942 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:03:20.860044 24455 catalog_manager.cc:1347] Generated new cluster ID: bbe5dcff7b0e4599b585949e56fd94b7
I20260928 22:03:20.860100 24455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:20.871726 24455 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:20.872349 24455 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:20.883358 24455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef955ac30a8b44a7a6609568bea4be73: Generated new TSK 0
I20260928 22:03:20.883627 24455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:20.886504 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24369
W20260928 22:03:20.891075 24354 connection.cc:535] client connection to 127.2.165.126:37661 recv error: Network error: failed to read from TLS socket (remote: 127.2.165.126:37661): Connection reset by peer (error 104)
2026-09-28T22:03:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:20 dist-test-slave-3t36 krb5kdc[24468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:20 dist-test-slave-3t36 krb5kdc[24468](info): set up 2 sockets
Sep 28 22:03:20 dist-test-slave-3t36 krb5kdc[24468](info): commencing operation
krb5kdc: starting...
W20260928 22:03:21.941005 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:22 dist-test-slave-3t36 krb5kdc[24468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:22.068367 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38627
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:22.149845 24481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:22.150059 24481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:22.150084 24481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:22.151269 24481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:22.151314 24481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:22.151328 24481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:22.151340 24481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:22.151352 24481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:22.152886 24481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:38627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:22.153149 24481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:22.153393 24481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:22.158371 24489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:22.158417 24487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:22.158481 24481 server_base.cc:1034] running on GCE node
W20260928 22:03:22.158380 24486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:22.158834 24481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:22.159122 24481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:22.160287 24481 hybrid_clock.cc:648] HybridClock initialized: now 1790633002160272 us; error 27 us; skew 500 ppm
Sep 28 22:03:22 dist-test-slave-3t36 krb5kdc[24468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633002, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:22.162950 24481 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:22.163491 24481 webserver.cc:460] Webserver started at http://127.2.165.126:43741/ using document root <none> and password file <none>
I20260928 22:03:22.163739 24481 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:22.163810 24481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:22.163955 24481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:22.165102 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "d25540edcf4a45ab886184eced5b80b5"
format_stamp: "Formatted at 2026-09-28 22:03:22 on dist-test-slave-3t36"
server_key: "13cc858f69d688010eae4f4ff78a57d0"
server_key_iv: "dab928d54332d65839bafcded666edcc"
server_key_version: "encryptionkey@0"
I20260928 22:03:22.165467 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "d25540edcf4a45ab886184eced5b80b5"
format_stamp: "Formatted at 2026-09-28 22:03:22 on dist-test-slave-3t36"
server_key: "13cc858f69d688010eae4f4ff78a57d0"
server_key_iv: "dab928d54332d65839bafcded666edcc"
server_key_version: "encryptionkey@0"
I20260928 22:03:22.166895 24481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 22:03:22.167722 24496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:22.167944 24481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:22.168054 24481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "d25540edcf4a45ab886184eced5b80b5"
format_stamp: "Formatted at 2026-09-28 22:03:22 on dist-test-slave-3t36"
server_key: "13cc858f69d688010eae4f4ff78a57d0"
server_key_iv: "dab928d54332d65839bafcded666edcc"
server_key_version: "encryptionkey@0"
I20260928 22:03:22.168140 24481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:22.171651 24481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:22.173841 24481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:22.174003 24481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:22.178220 24481 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:38627
I20260928 22:03:22.178251 24548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:38627 every 8 connection(s)
I20260928 22:03:22.178607 24481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:22.179554 24549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:22.181939 24549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5: Bootstrap starting.
I20260928 22:03:22.182538 24549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:22.182858 24549 log.cc:824] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:22.183578 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24481
I20260928 22:03:22.183710 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:22.183864 24549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5: No bootstrap required, opened a new log
I20260928 22:03:22.183921 2709 external_mini_cluster.cc:1428] Setting key 39e6afa543fca22b24846565dda07dfa
I20260928 22:03:22.186272 24549 raft_consensus.cc:348] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25540edcf4a45ab886184eced5b80b5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38627 } }
I20260928 22:03:22.186447 24549 raft_consensus.cc:374] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:22.186506 24549 raft_consensus.cc:729] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25540edcf4a45ab886184eced5b80b5, State: Initialized, Role: FOLLOWER
I20260928 22:03:22.186720 24549 consensus_queue.cc:260] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25540edcf4a45ab886184eced5b80b5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38627 } }
I20260928 22:03:22.186829 24549 raft_consensus.cc:388] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:22.186877 24549 raft_consensus.cc:482] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:22.186914 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:22.184484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43569 (local address 127.2.165.126:38627)
0928 22:03:22.184732 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:22.184809 (+ 77us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:22.184846 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:03:22.185269 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:22.185276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:22.185302 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:22.185456 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:22.186169 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:22.186174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:22.186174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:22.186178 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:22.186222 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:22.186233 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:22.186315 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:22.186384 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:03:22.186428 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260928 22:03:22.186931 24549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:22.187672 24549 raft_consensus.cc:504] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25540edcf4a45ab886184eced5b80b5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38627 } }
I20260928 22:03:22.187804 24549 leader_election.cc:302] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d25540edcf4a45ab886184eced5b80b5; no voters:
I20260928 22:03:22.187999 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:22.188011 24549 leader_election.cc:288] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:22.188030 24554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:22.188171 24554 raft_consensus.cc:686] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [term 1 LEADER]: Becoming Leader. State: Replica: d25540edcf4a45ab886184eced5b80b5, State: Running, Role: LEADER
I20260928 22:03:22.188305 24549 sys_catalog.cc:564] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:22.188339 24554 consensus_queue.cc:237] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25540edcf4a45ab886184eced5b80b5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38627 } }
I20260928 22:03:22.188798 24556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25540edcf4a45ab886184eced5b80b5. Latest consensus state: current_term: 1 leader_uuid: "d25540edcf4a45ab886184eced5b80b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25540edcf4a45ab886184eced5b80b5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38627 } } }
I20260928 22:03:22.188912 24556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:22.189180 24555 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d25540edcf4a45ab886184eced5b80b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25540edcf4a45ab886184eced5b80b5" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 38627 } } }
I20260928 22:03:22.189487 24555 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:22.189661 24562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:22.190264 24562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:22.191709 24562 catalog_manager.cc:1347] Generated new cluster ID: 1361199935ad4cd89d6fa25ea8c9a263
I20260928 22:03:22.191768 24562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:22.198436 24562 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:22.198678 24562 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:22.199783 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:22.193318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33377 (local address 127.2.165.126:38627)
0928 22:03:22.193360 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:22.193363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:22.198481 (+ 5118us) server_negotiation.cc:407] Connection header received
0928 22:03:22.198635 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:22.198636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:22.198666 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:22.198751 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:22.199539 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:22.199541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:22.199541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:22.199546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:22.199564 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:22.199568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:22.199634 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:22.199689 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 22:03:22.199703 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:03:22.212862 24562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d25540edcf4a45ab886184eced5b80b5: Generated new TSK 0
I20260928 22:03:22.213099 24562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:22.223470 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24481
2026-09-28T22:03:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:22 dist-test-slave-3t36 krb5kdc[24580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:22 dist-test-slave-3t36 krb5kdc[24580](info): set up 2 sockets
Sep 28 22:03:22 dist-test-slave-3t36 krb5kdc[24580](info): commencing operation
krb5kdc: starting...
W20260928 22:03:23.269117 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:23 dist-test-slave-3t36 krb5kdc[24580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:23.395359 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42599
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:23.477036 24593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:23.477248 24593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:23.477270 24593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:23.478379 24593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:23.478415 24593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:23.478430 24593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:23.478442 24593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:23.478453 24593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:23.479897 24593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:42599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:23.480094 24593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:23.480263 24593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:03:23.485380 24593 server_base.cc:1034] running on GCE node
W20260928 22:03:23.485381 24598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:23.485350 24599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:23.485350 24601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:23.485922 24593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:23.486233 24593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:23.487437 24593 hybrid_clock.cc:648] HybridClock initialized: now 1790633003487404 us; error 50 us; skew 500 ppm
Sep 28 22:03:23 dist-test-slave-3t36 krb5kdc[24580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633003, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:23.490144 24593 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:23.490671 24593 webserver.cc:460] Webserver started at http://127.2.165.126:37007/ using document root <none> and password file <none>
I20260928 22:03:23.490918 24593 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:23.490988 24593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:23.491149 24593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:23.492260 24593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "22fe3b8bf8a74ebcaa219aed704d30d4"
format_stamp: "Formatted at 2026-09-28 22:03:23 on dist-test-slave-3t36"
server_key: "d2c3e7d67416c9bfc6a014bdb3dd4840"
server_key_iv: "a11a75db9a38ca8419eb43358de9df10"
server_key_version: "encryptionkey@0"
I20260928 22:03:23.492638 24593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "22fe3b8bf8a74ebcaa219aed704d30d4"
format_stamp: "Formatted at 2026-09-28 22:03:23 on dist-test-slave-3t36"
server_key: "d2c3e7d67416c9bfc6a014bdb3dd4840"
server_key_iv: "a11a75db9a38ca8419eb43358de9df10"
server_key_version: "encryptionkey@0"
I20260928 22:03:23.494063 24593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:23.494820 24608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:23.494994 24593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:23.495098 24593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "22fe3b8bf8a74ebcaa219aed704d30d4"
format_stamp: "Formatted at 2026-09-28 22:03:23 on dist-test-slave-3t36"
server_key: "d2c3e7d67416c9bfc6a014bdb3dd4840"
server_key_iv: "a11a75db9a38ca8419eb43358de9df10"
server_key_version: "encryptionkey@0"
I20260928 22:03:23.495182 24593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:23.505144 24593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:23.507341 24593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:23.507525 24593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:23.512034 24593 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:42599
I20260928 22:03:23.512080 24660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:42599 every 8 connection(s)
I20260928 22:03:23.512445 24593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:23.513530 24661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:23.515882 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4: Bootstrap starting.
I20260928 22:03:23.516500 24661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:23.516834 24661 log.cc:824] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:23.517650 24661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4: No bootstrap required, opened a new log
I20260928 22:03:23.519429 24661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42599 } }
I20260928 22:03:23.519586 24661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:23.519609 24661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fe3b8bf8a74ebcaa219aed704d30d4, State: Initialized, Role: FOLLOWER
I20260928 22:03:23.519780 24661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42599 } }
I20260928 22:03:23.519846 24661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:23.519867 24661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:23.519896 24661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:23.520622 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24593
I20260928 22:03:23.520915 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:23.521056 24661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42599 } }
I20260928 22:03:23.521159 2709 external_mini_cluster.cc:1428] Setting key f8e9cdfc5e3ce395ec8a3e9799f7626a
I20260928 22:03:23.521162 24661 leader_election.cc:302] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22fe3b8bf8a74ebcaa219aed704d30d4; no voters:
I20260928 22:03:23.521358 24661 leader_election.cc:288] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:23.521571 24665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:23.521642 24661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:23.521749 24665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [term 1 LEADER]: Becoming Leader. State: Replica: 22fe3b8bf8a74ebcaa219aed704d30d4, State: Running, Role: LEADER
I20260928 22:03:23.521906 24665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42599 } }
I20260928 22:03:23.522348 24665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22fe3b8bf8a74ebcaa219aed704d30d4. Latest consensus state: current_term: 1 leader_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42599 } } }
I20260928 22:03:23.522377 24664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22fe3b8bf8a74ebcaa219aed704d30d4" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42599 } } }
I20260928 22:03:23.522521 24664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:23.522748 24665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:23.522810 24675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:23.523452 24675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:03:23 dist-test-slave-3t36 krb5kdc[24580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:23.529623 24675 catalog_manager.cc:1347] Generated new cluster ID: 67788ab4bd92492597df060df5a90c77
I20260928 22:03:23.529691 24675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:23.533100 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:23.521856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54279 (local address 127.2.165.126:42599)
0928 22:03:23.522022 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:23.522033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:23.522084 (+ 51us) server_negotiation.cc:407] Connection header received
0928 22:03:23.522204 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:23.522211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:23.522291 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:23.523406 (+ 1115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:23.523518 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:23.525003 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:23.526083 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:23.526544 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:23.526634 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:23.529518 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:23.529522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:23.529522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:23.529536 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:23.531454 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:23.531841 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:23.531844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:23.531844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:23.531959 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:23.532172 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:23.532173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:23.532173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:23.532443 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:23.532621 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:23.532759 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 22:03:23.532911 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260928 22:03:23.534209 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:23.547571 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:23.537089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53099 (local address 127.2.165.126:42599)
0928 22:03:23.537143 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:23.537146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:23.540922 (+ 3776us) server_negotiation.cc:407] Connection header received
0928 22:03:23.540996 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:23.540997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:23.541010 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:23.541087 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:23.541806 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:23.542402 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:23.543122 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:23.543343 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:23.543423 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:23.544841 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:23.544843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:23.544843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:23.544851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:23.546347 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:23.546715 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:23.546717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:23.546718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:23.546784 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:23.546989 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:23.546990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:23.546990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:23.547116 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:23.547232 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:23.547372 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 22:03:23.547450 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:03:23.550988 24675 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:23.551529 24675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:23.554646 24675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22fe3b8bf8a74ebcaa219aed704d30d4: Generated new TSK 0
I20260928 22:03:23.554863 24675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:23.559851 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24593
2026-09-28T22:03:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:23 dist-test-slave-3t36 krb5kdc[24692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:23 dist-test-slave-3t36 krb5kdc[24692](info): set up 2 sockets
Sep 28 22:03:23 dist-test-slave-3t36 krb5kdc[24692](info): commencing operation
krb5kdc: starting...
W20260928 22:03:24.613044 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:24 dist-test-slave-3t36 krb5kdc[24692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:24.740094 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41299
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:41299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:24.825238 24705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:24.825515 24705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:24.825558 24705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:24.826826 24705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:24.826910 24705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:24.826933 24705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:24.826956 24705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:24.826987 24705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:24.828596 24705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:41299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:24.828943 24705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:24.829197 24705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:24.834295 24713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:24.834307 24705 server_base.cc:1034] running on GCE node
W20260928 22:03:24.834316 24711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:24.834308 24710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:24.834797 24705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:24.835176 24705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:24.836365 24705 hybrid_clock.cc:648] HybridClock initialized: now 1790633004836323 us; error 58 us; skew 500 ppm
Sep 28 22:03:24 dist-test-slave-3t36 krb5kdc[24692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:24.850023 24705 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:24.850592 24705 webserver.cc:460] Webserver started at http://127.2.165.126:46121/ using document root <none> and password file <none>
I20260928 22:03:24.850865 24705 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:24.850940 24705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:24.851099 24705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:24.852231 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "8400199a178b474883b73a1f70c00127"
format_stamp: "Formatted at 2026-09-28 22:03:24 on dist-test-slave-3t36"
server_key: "311a2bcb644edc1735ca83b705bdf465"
server_key_iv: "e8ed00e47a2f1df41417e6edcd8d361c"
server_key_version: "encryptionkey@0"
I20260928 22:03:24.852609 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "8400199a178b474883b73a1f70c00127"
format_stamp: "Formatted at 2026-09-28 22:03:24 on dist-test-slave-3t36"
server_key: "311a2bcb644edc1735ca83b705bdf465"
server_key_iv: "e8ed00e47a2f1df41417e6edcd8d361c"
server_key_version: "encryptionkey@0"
I20260928 22:03:24.854079 24705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:24.854830 24720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:24.855096 24705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:24.855212 24705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "8400199a178b474883b73a1f70c00127"
format_stamp: "Formatted at 2026-09-28 22:03:24 on dist-test-slave-3t36"
server_key: "311a2bcb644edc1735ca83b705bdf465"
server_key_iv: "e8ed00e47a2f1df41417e6edcd8d361c"
server_key_version: "encryptionkey@0"
I20260928 22:03:24.855294 24705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:24.872071 24705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:24.874480 24705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:24.874670 24705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:24.878826 24705 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:41299
I20260928 22:03:24.878904 24772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:41299 every 8 connection(s)
I20260928 22:03:24.879280 24705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:24.880137 24773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:24.882445 24773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127: Bootstrap starting.
I20260928 22:03:24.882989 24773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:24.883246 24773 log.cc:824] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:24.883929 24773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127: No bootstrap required, opened a new log
I20260928 22:03:24.885505 24773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8400199a178b474883b73a1f70c00127" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41299 } }
I20260928 22:03:24.885654 24773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:24.885679 24773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8400199a178b474883b73a1f70c00127, State: Initialized, Role: FOLLOWER
I20260928 22:03:24.885823 24773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8400199a178b474883b73a1f70c00127" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41299 } }
I20260928 22:03:24.885913 24773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:24.885948 24773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:24.885993 24773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:24.885985 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24705
I20260928 22:03:24.886098 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:24.886309 2709 external_mini_cluster.cc:1428] Setting key 1b3001e14e64f63d1fe0a99d2f97de4f
I20260928 22:03:24.887161 24773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8400199a178b474883b73a1f70c00127" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41299 } }
I20260928 22:03:24.887251 24773 leader_election.cc:302] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8400199a178b474883b73a1f70c00127; no voters:
I20260928 22:03:24.887435 24773 leader_election.cc:288] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:24.887485 24778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:24.887669 24778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [term 1 LEADER]: Becoming Leader. State: Replica: 8400199a178b474883b73a1f70c00127, State: Running, Role: LEADER
I20260928 22:03:24.887735 24773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:24.887818 24778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8400199a178b474883b73a1f70c00127" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41299 } }
I20260928 22:03:24.889168 24780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8400199a178b474883b73a1f70c00127. Latest consensus state: current_term: 1 leader_uuid: "8400199a178b474883b73a1f70c00127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8400199a178b474883b73a1f70c00127" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41299 } } }
I20260928 22:03:24.889415 24780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:24.889637 24779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8400199a178b474883b73a1f70c00127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8400199a178b474883b73a1f70c00127" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 41299 } } }
I20260928 22:03:24.889712 24779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127 [sys.catalog]: This master's current role is: LEADER
W20260928 22:03:24.889788 24793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:24.889855 24793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:24.891708 24794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:24.891999 24794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:24.892596 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:24.886832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47479 (local address 127.2.165.126:41299)
0928 22:03:24.887048 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:24.887057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:24.887084 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:03:24.887183 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:24.887189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:24.887261 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:24.887458 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:24.887821 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:24.889431 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:24.890464 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:24.890960 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:24.891103 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:24.891840 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:24.891844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:24.891844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:24.892066 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:24.892112 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:24.892130 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:24.892202 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:24.892283 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 22:03:24.892355 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":118,"threads_started":1}
I20260928 22:03:24.893694 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:24.893734 24794 catalog_manager.cc:1347] Generated new cluster ID: 64033924db754a709123eab36bfa733e
I20260928 22:03:24.893771 24794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:24.902890 24794 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:24.903434 24794 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:24.907162 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:24.895199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59429 (local address 127.2.165.126:41299)
0928 22:03:24.895234 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:24.895235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:24.902530 (+ 7295us) server_negotiation.cc:407] Connection header received
0928 22:03:24.902641 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:24.902642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:24.902656 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:24.902736 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:24.903687 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:24.904601 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:24.905658 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:24.906035 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:24.906118 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:24.906874 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:24.906875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:24.906876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:24.906883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:24.906901 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:24.906906 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:24.906971 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:24.907022 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 22:03:24.907069 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:03:24.910887 24794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8400199a178b474883b73a1f70c00127: Generated new TSK 0
I20260928 22:03:24.911114 24794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:24.917405 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24705
2026-09-28T22:03:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:24 dist-test-slave-3t36 krb5kdc[24804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:24 dist-test-slave-3t36 krb5kdc[24804](info): set up 2 sockets
Sep 28 22:03:24 dist-test-slave-3t36 krb5kdc[24804](info): commencing operation
krb5kdc: starting...
W20260928 22:03:26.001243 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:26 dist-test-slave-3t36 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:26.129173 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42321
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:26.211872 24817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:26.212146 24817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:26.212188 24817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:26.213462 24817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:26.213538 24817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:26.213563 24817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:26.213585 24817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:26.213613 24817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:26.215168 24817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:42321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:26.215461 24817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:26.215701 24817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:26.220903 24822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:26.220908 24825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:26.220953 24823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:26.221189 24817 server_base.cc:1034] running on GCE node
I20260928 22:03:26.221513 24817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:26.221830 24817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:26.223035 24817 hybrid_clock.cc:648] HybridClock initialized: now 1790633006223014 us; error 66 us; skew 500 ppm
Sep 28 22:03:26 dist-test-slave-3t36 krb5kdc[24804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:26.226121 24817 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:26.226663 24817 webserver.cc:460] Webserver started at http://127.2.165.126:32915/ using document root <none> and password file <none>
I20260928 22:03:26.226907 24817 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:26.227025 24817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:26.227177 24817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:26.228317 24817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "d13f2181d38c40e584ace9e856e5f41d"
format_stamp: "Formatted at 2026-09-28 22:03:26 on dist-test-slave-3t36"
server_key: "628dbd3c96c30a59ad17f401d67cc7a5"
server_key_iv: "88efe6b6f7830ab04d85526d4b23def1"
server_key_version: "encryptionkey@0"
I20260928 22:03:26.228765 24817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "d13f2181d38c40e584ace9e856e5f41d"
format_stamp: "Formatted at 2026-09-28 22:03:26 on dist-test-slave-3t36"
server_key: "628dbd3c96c30a59ad17f401d67cc7a5"
server_key_iv: "88efe6b6f7830ab04d85526d4b23def1"
server_key_version: "encryptionkey@0"
I20260928 22:03:26.230268 24817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:26.231285 24832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:26.231515 24817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:26.231630 24817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "d13f2181d38c40e584ace9e856e5f41d"
format_stamp: "Formatted at 2026-09-28 22:03:26 on dist-test-slave-3t36"
server_key: "628dbd3c96c30a59ad17f401d67cc7a5"
server_key_iv: "88efe6b6f7830ab04d85526d4b23def1"
server_key_version: "encryptionkey@0"
I20260928 22:03:26.231716 24817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:26.235255 24817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:26.237785 24817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:26.237967 24817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:26.242084 24817 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:42321
I20260928 22:03:26.242134 24884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:42321 every 8 connection(s)
I20260928 22:03:26.242515 24817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:26.243578 24885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:26.244278 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24817
I20260928 22:03:26.244411 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:26.244602 2709 external_mini_cluster.cc:1428] Setting key 48a79716bce92073873dde2bfc56ed8f
I20260928 22:03:26.246973 24885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d: Bootstrap starting.
I20260928 22:03:26.247148 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:26.245173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49469 (local address 127.2.165.126:42321)
0928 22:03:26.245332 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:26.245368 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:26.245395 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:03:26.245787 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:26.245793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:26.245818 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:26.245961 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:26.246603 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:26.246606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:26.246606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:26.246610 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:26.246647 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:26.246657 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:26.246712 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:26.246801 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:03:26.246890 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260928 22:03:26.247776 24885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:26.248117 24885 log.cc:824] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:26.248266 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:26.248931 24885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d: No bootstrap required, opened a new log
I20260928 22:03:26.250820 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:26.249781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48069 (local address 127.2.165.126:42321)
0928 22:03:26.249820 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:26.249822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:26.249888 (+ 66us) server_negotiation.cc:407] Connection header received
0928 22:03:26.249956 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:26.249956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:26.249959 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:26.250003 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:26.250550 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:26.250551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:26.250551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:03:26.250556 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:26.250573 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:03:26.250578 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:26.250637 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:26.250714 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 22:03:26.250726 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:03:26.251374 24885 raft_consensus.cc:348] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f2181d38c40e584ace9e856e5f41d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42321 } }
I20260928 22:03:26.251528 24885 raft_consensus.cc:374] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:03:26.251551 24885 raft_consensus.cc:729] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13f2181d38c40e584ace9e856e5f41d, State: Initialized, Role: FOLLOWER
I20260928 22:03:26.251710 24885 consensus_queue.cc:260] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f2181d38c40e584ace9e856e5f41d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42321 } }
I20260928 22:03:26.251798 24885 raft_consensus.cc:388] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:03:26.251854 24885 raft_consensus.cc:482] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:03:26.251909 24885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:26.252496 24885 raft_consensus.cc:504] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f2181d38c40e584ace9e856e5f41d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42321 } }
I20260928 22:03:26.252621 24885 leader_election.cc:302] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d13f2181d38c40e584ace9e856e5f41d; no voters:
I20260928 22:03:26.252847 24885 leader_election.cc:288] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:03:26.252879 24895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:26.252995 24895 raft_consensus.cc:686] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [term 1 LEADER]: Becoming Leader. State: Replica: d13f2181d38c40e584ace9e856e5f41d, State: Running, Role: LEADER
I20260928 22:03:26.253086 24895 consensus_queue.cc:237] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f2181d38c40e584ace9e856e5f41d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42321 } }
I20260928 22:03:26.253125 24885 sys_catalog.cc:564] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:26.253460 24896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d13f2181d38c40e584ace9e856e5f41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f2181d38c40e584ace9e856e5f41d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42321 } } }
I20260928 22:03:26.253574 24896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:26.253772 24897 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d13f2181d38c40e584ace9e856e5f41d. Latest consensus state: current_term: 1 leader_uuid: "d13f2181d38c40e584ace9e856e5f41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f2181d38c40e584ace9e856e5f41d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 42321 } } }
I20260928 22:03:26.253849 24897 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:26.254289 24901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:26.254760 24901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:26.256443 24901 catalog_manager.cc:1347] Generated new cluster ID: 8d9a54b4406546a6bfad63817e08f2c6
I20260928 22:03:26.256502 24901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:26.269904 24901 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:26.270147 24901 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:26.277143 24901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d13f2181d38c40e584ace9e856e5f41d: Generated new TSK 0
I20260928 22:03:26.277401 24901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:26.288002 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24817
2026-09-28T22:03:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1370 ms)
[----------] 18 tests from AuthTokenIssuingTest (24347 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:26 dist-test-slave-3t36 krb5kdc[24916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:26 dist-test-slave-3t36 krb5kdc[24916](info): set up 2 sockets
Sep 28 22:03:26 dist-test-slave-3t36 krb5kdc[24916](info): commencing operation
krb5kdc: starting...
W20260928 22:03:27.353319 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:27.480271 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:40175
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:40175,127.2.165.125:41957,127.2.165.124:39355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:27.563171 24929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:03:27.563467 24929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:27.563498 24929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:27.563542 24929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:27.564791 24929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:27.564873 24929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:27.564895 24929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:27.564917 24929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:27.564945 24929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:27.566550 24929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:40175,127.2.165.125:41957,127.2.165.124:39355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:27.566864 24929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:27.567108 24929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:27.572486 24937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:27.572520 24935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:27.572486 24934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:27.572780 24929 server_base.cc:1034] running on GCE node
I20260928 22:03:27.573031 24929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:27.573338 24929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:27.574561 24929 hybrid_clock.cc:648] HybridClock initialized: now 1790633007574544 us; error 41 us; skew 500 ppm
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:27.578388 24929 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:27.578944 24929 webserver.cc:460] Webserver started at http://127.2.165.126:38647/ using document root <none> and password file <none>
I20260928 22:03:27.579209 24929 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:27.579280 24929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:27.579424 24929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:27.580617 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "dd875ae4792349cdbe70c2b97671749d"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "7880fd32f9535c967d3f2845dd00ca89"
server_key_iv: "e34579e84b0dd91ff5e9a1ef9988e0ab"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.581068 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "dd875ae4792349cdbe70c2b97671749d"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "7880fd32f9535c967d3f2845dd00ca89"
server_key_iv: "e34579e84b0dd91ff5e9a1ef9988e0ab"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.582500 24929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:27.583429 24944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:27.583695 24929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:27.583868 24929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "dd875ae4792349cdbe70c2b97671749d"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "7880fd32f9535c967d3f2845dd00ca89"
server_key_iv: "e34579e84b0dd91ff5e9a1ef9988e0ab"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.583971 24929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:27.597450 24929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:27.600154 24929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:27.600390 24929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:27.604722 24929 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:40175
I20260928 22:03:27.604832 24996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:40175 every 8 connection(s)
I20260928 22:03:27.605203 24929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:27.605628 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 24929
I20260928 22:03:27.605773 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:27.605979 2709 external_mini_cluster.cc:1428] Setting key 52aad718d37976bc5715026ff72ae0a3
I20260928 22:03:27.606487 24997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:27.608863 24997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.165.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.125@KRBTEST.COM" created.
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:27.622306 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.609610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40175 (local address 127.0.0.1:57182)
0928 22:03:27.609821 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.609880 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.610635 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.611212 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.611215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.611623 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.611892 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.611899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.613225 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.613226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.614067 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.614069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.614653 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.614655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.614725 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.615455 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.615465 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.617744 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.620035 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.620045 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.620048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.620382 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.620664 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.620665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.620666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.620841 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.621470 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.621478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.621583 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.621717 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.621975 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260928 22:03:27.622310 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.609794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.2.165.126:40175)
0928 22:03:27.609915 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.609920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.609981 (+ 61us) server_negotiation.cc:407] Connection header received
0928 22:03:27.610889 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.610892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.610913 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.611079 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.611996 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.613122 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.614161 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.614520 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.614666 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.617849 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.617859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.617860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.617869 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.619931 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.620487 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.620491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.620491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.620583 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.620971 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.620972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.620973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.621377 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.621460 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.621801 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.622090 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260928 22:03:27.623389 24997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:27.623836 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.623663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47010)
0928 22:03:27.623721 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.623794 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:03:27.624029 24948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
I20260928 22:03:27.624531 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.624411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47026)
0928 22:03:27.624462 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.624499 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:03:27.624662 24997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:41957: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab.
Entry for principal kudu/127.2.165.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab.
I20260928 22:03:27.636335 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:41957
--webserver_interface=127.2.165.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab
--principal=kudu/127.2.165.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:40175,127.2.165.125:41957,127.2.165.124:39355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:03:27.673956 24997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } attempt: 1
I20260928 22:03:27.674540 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.674369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47034)
0928 22:03:27.674445 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.674497 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:03:27.674934 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.674837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47046)
0928 22:03:27.674881 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.674911 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:03:27.675101 24997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:41957: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
W20260928 22:03:27.722756 25004 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:03:27.723024 25004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:27.723065 25004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:27.723105 25004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:27.724287 25004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:27.724352 25004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:27.724378 25004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:27.724409 25004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:27.724435 25004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:27.726024 25004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:40175,127.2.165.125:41957,127.2.165.124:39355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab
--principal=kudu/127.2.165.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.165.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:27.726307 25004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:27.726532 25004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:27.731565 25009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:27.731560 25010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:27.731868 25004 server_base.cc:1034] running on GCE node
W20260928 22:03:27.731575 25012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:27.732179 25004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:27.732487 25004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:27.733651 25004 hybrid_clock.cc:648] HybridClock initialized: now 1790633007733621 us; error 49 us; skew 500 ppm
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:27.736392 25004 init.cc:377] Logged in from keytab as kudu/127.2.165.125@KRBTEST.COM (short username kudu)
I20260928 22:03:27.736935 25004 webserver.cc:460] Webserver started at http://127.2.165.125:42465/ using document root <none> and password file <none>
I20260928 22:03:27.737141 25004 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:27.737185 25004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:27.737273 25004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:27.738296 25004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data/instance:
uuid: "5c1f73ce1749429f90fc7fbb5b614c9f"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "824537ccfed8c991220f59464a612b0b"
server_key_iv: "a8d16b6297e0cb9c28879113139757df"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.738626 25004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/wal/instance:
uuid: "5c1f73ce1749429f90fc7fbb5b614c9f"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "824537ccfed8c991220f59464a612b0b"
server_key_iv: "a8d16b6297e0cb9c28879113139757df"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.740082 25004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:03:27.741003 25019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:27.741195 25004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:27.741286 25004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/wal
uuid: "5c1f73ce1749429f90fc7fbb5b614c9f"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "824537ccfed8c991220f59464a612b0b"
server_key_iv: "a8d16b6297e0cb9c28879113139757df"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.741346 25004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:27.743283 24997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } attempt: 2
I20260928 22:03:27.743707 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.743581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47058)
0928 22:03:27.743637 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.743669 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:03:27.744125 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.744040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47068)
0928 22:03:27.744083 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.744108 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:03:27.744227 24997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:41957: Network error: Client connection negotiation failed: client connection to 127.2.165.125:41957: connect: Connection refused (error 111)
I20260928 22:03:27.761279 25004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:27.763628 25004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:27.763857 25004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:27.768039 25004 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.125:41957
I20260928 22:03:27.768119 25071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.125:41957 every 8 connection(s)
I20260928 22:03:27.768457 25004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
I20260928 22:03:27.769481 25072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:27.771128 25072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:27.771803 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 25004
I20260928 22:03:27.771939 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-1/wal/instance
I20260928 22:03:27.772125 2709 external_mini_cluster.cc:1428] Setting key a86f1de6d4f2e3bb0825736c604b0121
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.165.124@KRBTEST.COM; defaulting to no policy
I20260928 22:03:27.781366 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.771912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.2.165.126:40175)
0928 22:03:27.771965 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.771968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.772177 (+ 209us) server_negotiation.cc:407] Connection header received
0928 22:03:27.772751 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.772752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.772769 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.772830 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.773506 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.774008 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.775047 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.775391 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.775458 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.778025 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.778027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.778027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.778037 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.779747 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.780222 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.780223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.780224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.780290 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.780542 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.780543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.780543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.780733 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.780814 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.781173 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.781293 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:03:27.781549 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.771853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40175 (local address 127.0.0.1:57194)
0928 22:03:27.772038 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.772081 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.772605 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.772909 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.772912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.773175 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.773377 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.773381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.774114 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.774115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.774947 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.774949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.775517 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.775518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.775652 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.776250 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.776263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.777919 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.779841 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.779849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.779851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.780136 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.780369 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.780370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.780370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.780480 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.780846 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.780853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.780994 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.781100 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.781261 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260928 22:03:27.782220 25072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.124@KRBTEST.COM" created.
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:03:27.791095 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.782485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47078)
0928 22:03:27.782538 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.782551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.782598 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.782958 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.782959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.783107 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.783157 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.783158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.784461 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.784462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.785239 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.785241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.785714 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.785715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.785799 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.786287 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.786292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.787752 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.789863 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.789864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.789864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.790049 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.790283 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.790284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.790285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.790345 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.790677 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.790677 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.790720 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.790816 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.790934 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:03:27.791111 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.782553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.2.165.125:41957)
0928 22:03:27.782764 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.782768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.782784 (+ 16us) server_negotiation.cc:407] Connection header received
0928 22:03:27.782802 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.782804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.782815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.782955 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.783245 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.784352 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.785324 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.785636 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.785697 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.787842 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.787852 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.787852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.787861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.789746 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.790148 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.790151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.790152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.790215 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.790424 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.790426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.790426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.790594 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.790686 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.790855 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.790992 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":185,"thread_start_us":148,"threads_started":1}
I20260928 22:03:27.791800 25072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:27.792259 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.792128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35428)
0928 22:03:27.792179 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.792228 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:03:27.792447 25020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
I20260928 22:03:27.792946 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.792860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35432)
0928 22:03:27.792901 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.792928 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:03:27.793107 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:39355: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab.
Entry for principal kudu/127.2.165.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab.
I20260928 22:03:27.800282 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:39355
--webserver_interface=127.2.165.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab
--principal=kudu/127.2.165.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:40175,127.2.165.125:41957,127.2.165.124:39355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:03:27.835392 24997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } attempt: 3
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:03:27.842357 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } attempt: 1
I20260928 22:03:27.842805 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.842641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35438)
0928 22:03:27.842714 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.842749 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 22:03:27.843137 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.843039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35448)
0928 22:03:27.843086 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.843111 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:03:27.843189 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.835800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47094)
0928 22:03:27.835876 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.835889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.835960 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.836107 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.836108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.836280 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.836348 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.836350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.837145 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.837146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.837681 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.837683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.838080 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.838081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.838163 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.838701 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.838706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.840183 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.842039 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.842040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.842041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.842230 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.842461 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.842462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.842463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.842532 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.842842 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.842844 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.842906 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.842985 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.843092 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 22:03:27.843485 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.835883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.2.165.125:41957)
0928 22:03:27.835950 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.835952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.835962 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:03:27.836014 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.836015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.836028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.836099 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.836433 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.837046 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.837772 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.837991 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.838078 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.840312 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.840313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.840314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.840327 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.841919 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.842327 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.842329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.842329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.842394 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.842601 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.842602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.842602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.842765 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.842835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.843319 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.843407 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 22:03:27.843487 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:39355: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
I20260928 22:03:27.843951 24997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:27.844285 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.844203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35464)
0928 22:03:27.844236 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.844265 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 22:03:27.844592 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.844532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35476)
0928 22:03:27.844558 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.844579 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 22:03:27.844687 24997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:39355: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
I20260928 22:03:27.875881 24997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } attempt: 1
I20260928 22:03:27.876353 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.876216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35488)
0928 22:03:27.876293 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.876324 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:03:27.876663 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.876577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35496)
0928 22:03:27.876617 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.876641 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:03:27.876788 24997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:39355: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
W20260928 22:03:27.885416 25079 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:03:27.885658 25079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:27.885697 25079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:27.885747 25079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:27.886950 25079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:27.887009 25079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:27.887035 25079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:27.887076 25079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:27.887107 25079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:27.888638 25079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:40175,127.2.165.125:41957,127.2.165.124:39355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab
--principal=kudu/127.2.165.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.165.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:27.888942 25079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:27.889176 25079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:27.894062 25087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:27.894052 25084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:27.894045 25085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:27.894375 25079 server_base.cc:1034] running on GCE node
I20260928 22:03:27.894636 25079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:27.894999 25079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:27.896202 25079 hybrid_clock.cc:648] HybridClock initialized: now 1790633007896174 us; error 52 us; skew 500 ppm
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:27.898932 25079 init.cc:377] Logged in from keytab as kudu/127.2.165.124@KRBTEST.COM (short username kudu)
I20260928 22:03:27.899389 25079 webserver.cc:460] Webserver started at http://127.2.165.124:40607/ using document root <none> and password file <none>
I20260928 22:03:27.899638 25079 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:27.899703 25079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:27.899856 25079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:27.900983 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data/instance:
uuid: "5b9a38511fca42b1a3fd64044a3d0246"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "81fce5d16fac641abf987b51d7a3ff99"
server_key_iv: "6d2f10e4e28a66b07f55acb7f8799c51"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.901327 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/wal/instance:
uuid: "5b9a38511fca42b1a3fd64044a3d0246"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "81fce5d16fac641abf987b51d7a3ff99"
server_key_iv: "6d2f10e4e28a66b07f55acb7f8799c51"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.902705 25079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:03:27.903564 25094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:27.903728 25079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:27.903890 25079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/wal
uuid: "5b9a38511fca42b1a3fd64044a3d0246"
format_stamp: "Formatted at 2026-09-28 22:03:27 on dist-test-slave-3t36"
server_key: "81fce5d16fac641abf987b51d7a3ff99"
server_key_iv: "6d2f10e4e28a66b07f55acb7f8799c51"
server_key_version: "encryptionkey@0"
I20260928 22:03:27.903967 25079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:27.911660 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } attempt: 2
I20260928 22:03:27.912180 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.912051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35506)
0928 22:03:27.912109 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.912149 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:03:27.912580 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.912481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35518)
0928 22:03:27.912527 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.912558 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:03:27.912720 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:39355: Network error: Client connection negotiation failed: client connection to 127.2.165.124:39355: connect: Connection refused (error 111)
I20260928 22:03:27.923297 25079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:27.925714 25079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:27.925868 25079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:27.930028 25079 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.124:39355
I20260928 22:03:27.930092 25146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.124:39355 every 8 connection(s)
I20260928 22:03:27.930470 25079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
I20260928 22:03:27.931530 25147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:27.933230 25147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:27.935741 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 25079
I20260928 22:03:27.935858 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632730261496-2709-0/minicluster-data/master-2/wal/instance
I20260928 22:03:27.936041 2709 external_mini_cluster.cc:1428] Setting key abd6cffb45864e3095b2517bfd89d5b3
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:03:27.944455 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.933974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.2.165.126:40175)
0928 22:03:27.934027 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.934029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.934190 (+ 161us) server_negotiation.cc:407] Connection header received
0928 22:03:27.934665 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.934666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.934678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.934886 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.935522 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.936317 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.937994 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.938201 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.938275 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.940882 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.940884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.940884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.940891 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.942512 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.943262 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.943264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.943264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.943326 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.943682 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.943684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.943684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.943835 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.943915 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.944260 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.944352 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:03:27.944640 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.933902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:40175 (local address 127.0.0.1:57200)
0928 22:03:27.934063 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.934105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.934549 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.934962 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.934965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.935214 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.935420 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.935424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.936594 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.936595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.937898 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.937902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.938310 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.938311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.938436 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.939038 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.939050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.940797 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.942582 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.942590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.942592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.943150 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.943457 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.943458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.943459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.943579 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.943907 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.943914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.944081 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.944221 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.944386 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260928 22:03:27.945358 25147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:27.946473 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.936422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.2.165.124:39355)
0928 22:03:27.936597 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.936601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.936637 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:03:27.936725 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.936728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.936749 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.936815 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.937649 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.938509 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.939555 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.939914 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.940038 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.942944 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.942958 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.942958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.942966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.945148 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.945698 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.945701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.945702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.945767 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.945957 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.945957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.945958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.946142 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.946268 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.946345 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.946401 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:03:27.951962 24997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } attempt: 2
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:27.953269 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.945699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.2.165.125:41957)
0928 22:03:27.945743 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.945744 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.945753 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:03:27.945776 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.945777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.945791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.945861 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.946147 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.946845 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.947636 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.947856 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.947943 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.950089 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.950090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.950090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.950097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.951927 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.952340 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.952342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.952342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.952408 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.952658 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.952659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.952659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.952838 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.952907 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.953089 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.953171 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:03:27.953310 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.945624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:41957 (local address 127.0.0.1:47106)
0928 22:03:27.945672 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.945684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.945720 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.945870 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.945871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.946032 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.946075 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.946077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.946928 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.946929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.947563 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.947564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.947995 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.947996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.948057 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.948419 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.948424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.950001 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.952012 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.952013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.952014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.952246 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.952484 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.952484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.952485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.952540 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.952915 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.952916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.952968 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.953053 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.953153 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:03:27.954054 25147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:27.956238 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.947359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.2.165.126:40175)
0928 22:03:27.947396 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.947397 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.947404 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:03:27.947424 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.947424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.947432 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.947494 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.947920 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.948579 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.949854 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.950158 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.950226 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.953120 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.953122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.953122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.953130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.954826 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.955411 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.955412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.955412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.955467 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.955687 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.955688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.955689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.955799 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.956000 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.956040 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.956119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:03:27.960350 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.952233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35532)
0928 22:03:27.952287 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.952298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.952334 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.952465 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.952466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.952622 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.952666 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.952690 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.953582 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.953584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.954395 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.954399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.954901 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.954902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.954976 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.955507 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.955513 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.956937 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.958840 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.958842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.958843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.959076 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.959471 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.959473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.959474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.959551 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.959943 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.959945 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.960012 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.960085 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.960208 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:03:27.960367 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.952317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.2.165.124:39355)
0928 22:03:27.952366 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.952368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.952375 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:03:27.952394 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.952395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.952406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.952463 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.952760 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.953467 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.954492 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.954818 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.954898 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.957099 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.957100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.957100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.957108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.958737 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.959224 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.959226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.959226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.959290 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.959634 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.959635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.959636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.959882 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.959950 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.960133 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.960230 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 22:03:27 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:03:27.962941 24997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d: Bootstrap starting.
I20260928 22:03:27.963876 24997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:27.963908 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.954333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35536)
0928 22:03:27.954558 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:27.954568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:27.954600 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:27.954739 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:27.954740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:27.954898 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:27.954933 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.954934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.955918 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.955920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.956936 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:27.956938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.957404 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.957405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:27.957453 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.958677 (+ 1224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:27.958683 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:27.960301 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:27.962725 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.962727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.962727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.962912 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.963116 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:27.963117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:27.963118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:27.963172 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:27.963550 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:27.963552 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:27.963605 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 22:03:27.963679 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:03:27.963789 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260928 22:03:27.964241 24997 log.cc:824] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:27.964807 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.954409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.2.165.124:39355)
0928 22:03:27.954607 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.954609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.954620 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:03:27.954643 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.954644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.954655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.954733 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.954993 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.955822 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.957026 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.957333 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.957409 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.960393 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.960395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.960396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.960403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.962592 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.963001 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.963003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.963003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.963063 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.963240 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.963241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.963241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.963457 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.963536 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.964020 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.964112 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":121,"threads_started":1}
I20260928 22:03:27.965344 24997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d: No bootstrap required, opened a new log
I20260928 22:03:27.966392 25147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: Bootstrap starting.
I20260928 22:03:27.967206 25147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:27.967587 25147 log.cc:824] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:27.967706 24997 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:27.967885 24997 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd875ae4792349cdbe70c2b97671749d, State: Initialized, Role: FOLLOWER
I20260928 22:03:27.967981 24997 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:27.968305 25155 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:27.968361 24997 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:27.968431 25155 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:27.969100 25147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: No bootstrap required, opened a new log
I20260928 22:03:27.969759 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.957050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.2.165.125:41957)
0928 22:03:27.957092 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.957093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.957099 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:03:27.957118 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.957118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.957126 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.957182 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.957801 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.958541 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.959341 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.959742 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.959824 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.963388 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.963389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.963390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.963400 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.968040 (+ 4640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.968831 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.968833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.968833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.968907 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.969162 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.969164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.969164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.969338 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.969527 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.969565 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.969652 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:03:27.971130 25166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:27.971195 25166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:27.971776 25147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:27.971956 25147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9a38511fca42b1a3fd64044a3d0246, State: Initialized, Role: FOLLOWER
I20260928 22:03:27.972038 25147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:27.972376 25147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:27.972349 25168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:27.972539 25168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:27.972745 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:03:27.974104 24971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dd875ae4792349cdbe70c2b97671749d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57216
I20260928 22:03:27.974255 24971 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:03:27.974309 24971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:27.971626 24929 master_runner.cc:186] Error getting master registration for 127.2.165.125:41957: OK, instance_id { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" instance_seqno: 1790633007766783 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:03:27.975020 24929 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:03:27.975340 24971 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:27.975638 24971 leader_election.cc:288] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1f73ce1749429f90fc7fbb5b614c9f (127.2.165.125:41957), 5b9a38511fca42b1a3fd64044a3d0246 (127.2.165.124:39355)
I20260928 22:03:27.976347 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd875ae4792349cdbe70c2b97671749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5b9a38511fca42b1a3fd64044a3d0246"
I20260928 22:03:27.977630 25121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:27.978408 25121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd875ae4792349cdbe70c2b97671749d in term 1.
W20260928 22:03:27.976547 25179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 22:03:27.978689 24945 leader_election.cc:302] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9a38511fca42b1a3fd64044a3d0246, dd875ae4792349cdbe70c2b97671749d; no voters:
W20260928 22:03:27.978754 25179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:27.978909 25155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:27.979099 25155 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [term 1 LEADER]: Becoming Leader. State: Replica: dd875ae4792349cdbe70c2b97671749d, State: Running, Role: LEADER
I20260928 22:03:27.979218 25155 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:27.980026 25155 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd875ae4792349cdbe70c2b97671749d. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:27.980140 25155 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:27.981281 25079 master_runner.cc:186] Error getting master registration for 127.2.165.125:41957: OK, instance_id { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" instance_seqno: 1790633007766783 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:03:27.984138 25079 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:03:27.984494 25189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:27.984755 25189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:27.991907 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.979290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.2.165.124:39355)
0928 22:03:27.979334 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.979336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.980631 (+ 1295us) server_negotiation.cc:407] Connection header received
0928 22:03:27.980846 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.980847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.980884 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.980916 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.982320 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.983085 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.984976 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.985271 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.985317 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.987510 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.987512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.987512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.987521 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.989394 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.990761 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.990763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.990763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.990829 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.991097 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.991098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.991099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.991258 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.991378 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.991686 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.991780 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:03:27.995002 25121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [term 1 FOLLOWER]: Refusing update from remote peer dd875ae4792349cdbe70c2b97671749d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:27.995417 25185 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:27.997093 25168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd875ae4792349cdbe70c2b97671749d. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:27.997201 25168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:27.998171 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.980285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.2.165.125:41957)
0928 22:03:27.980341 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.980343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.986364 (+ 6021us) server_negotiation.cc:407] Connection header received
0928 22:03:27.986474 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.986476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.986489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.986523 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.988373 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.989399 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.990406 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.990705 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.990770 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:27.993262 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:27.993263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:27.993263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:27.993272 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:27.995048 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.997309 (+ 2261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.997310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.997310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.997384 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:27.997590 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:27.997590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:27.997591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:27.997714 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:27.997791 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:27.997982 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 22:03:27.998073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:03:27.999956 25185 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:28.000078 25185 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:28.000361 25168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:28.000452 25168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:28.001909 25191 mvcc.cc:205] Tried to move back new op lower bound from 7334432800742813696 to 7334432800685096960. Current Snapshot: MvccSnapshot[applied={T|T < 7334432800742813696}]
I20260928 22:03:28.002725 25192 mvcc.cc:205] Tried to move back new op lower bound from 7334432800742813696 to 7334432800685096960. Current Snapshot: MvccSnapshot[applied={T|T < 7334432800742813696 or (T in {7334432800742813696})}]
I20260928 22:03:28.003924 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } attempt: 3
I20260928 22:03:28.005468 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:27.979570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.2.165.126:40175)
0928 22:03:27.979632 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:27.979635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:27.986171 (+ 6536us) server_negotiation.cc:407] Connection header received
0928 22:03:27.986253 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:27.986254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:27.986269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:27.986347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:27.997163 (+ 10816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.997907 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.998997 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:27.999301 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:27.999375 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:28.001732 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:28.001733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:28.001734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:28.001742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:28.003531 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:28.004119 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:28.004136 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:28.004136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:28.004213 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:28.004592 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:28.004593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:28.004593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:28.004759 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:28.004868 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:28.005272 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 22:03:28.005366 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:03:28.006294 25189 catalog_manager.cc:1347] Generated new cluster ID: 65d454a58b08454dbdb60b0ac89a1743
I20260928 22:03:28.006346 25189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:03:28 dist-test-slave-3t36 krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:03:28.012125 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:28.004214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:39355 (local address 127.0.0.1:35558)
0928 22:03:28.004330 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:28.004341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:28.004380 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:28.004520 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:28.004520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:28.004734 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:28.004798 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:28.004800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:28.005749 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:28.005750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:28.006522 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:28.006525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:28.006992 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:28.006993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:28.007063 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:28.007550 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:28.007555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:28.009089 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:28.011059 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:28.011060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:28.011060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:28.011279 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:28.011486 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:28.011486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:28.011487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:28.011538 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:28.011808 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:28.011808 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:28.011866 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 22:03:28.011926 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:03:28.012021 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 22:03:28.012136 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:28.004347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.2.165.124:39355)
0928 22:03:28.004394 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.004396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.004406 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:03:28.004431 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.004432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.004442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.004512 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.004861 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:28.005662 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:28.006598 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:28.006913 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:28.006988 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:28.009162 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:28.009163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:28.009163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:28.009172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:28.010951 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:28.011346 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:28.011347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:28.011348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:28.011412 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:28.011582 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:28.011583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:28.011583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:28.011720 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:28.011792 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:28.011953 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:03:28.012037 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:03:28.013903 25072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: Bootstrap starting.
I20260928 22:03:28.014647 25072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:28.014938 25072 log.cc:824] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:28.015868 25072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: No bootstrap required, opened a new log
I20260928 22:03:28.018002 25072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:28.018149 25072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1f73ce1749429f90fc7fbb5b614c9f, State: Initialized, Role: FOLLOWER
I20260928 22:03:28.018226 25072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } }
I20260928 22:03:28.018537 25072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:28.019889 25045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:28.019743 25046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd875ae4792349cdbe70c2b97671749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f"
I20260928 22:03:28.020807 25045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [term 1 FOLLOWER]: Refusing update from remote peer dd875ae4792349cdbe70c2b97671749d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:28.021127 25155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:28.022337 25046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd875ae4792349cdbe70c2b97671749d in term 1.
I20260928 22:03:28.023751 25212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:28.023936 25212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:28.024195 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd875ae4792349cdbe70c2b97671749d. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:28.024272 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:28.024219 25197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:28.024331 25197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:03:28.024848 25208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:28.024904 25208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:28.029877 25189 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:28.030516 25189 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:28.034497 25185 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd875ae4792349cdbe70c2b97671749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd875ae4792349cdbe70c2b97671749d" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 40175 } } peers { permanent_uuid: "5c1f73ce1749429f90fc7fbb5b614c9f" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 41957 } } peers { permanent_uuid: "5b9a38511fca42b1a3fd64044a3d0246" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 39355 } } }
I20260928 22:03:28.034605 25185 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:28.047835 25189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd875ae4792349cdbe70c2b97671749d: Generated new TSK 0
I20260928 22:03:28.048100 25189 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:03:28.085593 25001 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.082711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.2.165.126:40175)
0928 22:03:28.082797 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.082799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.083737 (+ 938us) server_negotiation.cc:407] Connection header received
0928 22:03:28.083917 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.083918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.083931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.083966 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.085472 (+ 1506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57236: BlockingRecv error: recv got EOF from 127.0.0.1:57236 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 22:03:28.086462 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.083247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.2.165.125:41957)
0928 22:03:28.083332 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.083334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.084659 (+ 1325us) server_negotiation.cc:407] Connection header received
0928 22:03:28.084925 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.084926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.084939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.084973 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.086351 (+ 1378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47124: BlockingRecv error: recv got EOF from 127.0.0.1:47124 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 22:03:28.088153 25152 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.087132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.2.165.124:39355)
0928 22:03:28.087186 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.087188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.087341 (+ 153us) server_negotiation.cc:407] Connection header received
0928 22:03:28.087478 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.087479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.087491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.087524 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.088034 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35568: BlockingRecv error: recv got EOF from 127.0.0.1:35568 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 22:03:28.094164 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.093245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.2.165.125:41957)
0928 22:03:28.093306 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.093308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.093524 (+ 216us) server_negotiation.cc:407] Connection header received
0928 22:03:28.093641 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.093642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.093663 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.093693 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.094107 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47130: BlockingRecv error: recv got EOF from 127.0.0.1:47130 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 22:03:28.097589 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.096753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.2.165.125:41957)
0928 22:03:28.096790 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.096791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.097041 (+ 250us) server_negotiation.cc:407] Connection header received
0928 22:03:28.097140 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.097141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.097152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.097180 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.097534 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47146: BlockingRecv error: recv got EOF from 127.0.0.1:47146 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:03:28.101815 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.101015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.2.165.125:41957)
0928 22:03:28.101053 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.101056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.101291 (+ 235us) server_negotiation.cc:407] Connection header received
0928 22:03:28.101392 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.101393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.101403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.101433 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.101760 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47150: BlockingRecv error: recv got EOF from 127.0.0.1:47150 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 22:03:28.108318 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.107382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.2.165.125:41957)
0928 22:03:28.107445 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.107447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.107721 (+ 274us) server_negotiation.cc:407] Connection header received
0928 22:03:28.107817 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.107817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.107828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.107856 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.108255 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47160: BlockingRecv error: recv got EOF from 127.0.0.1:47160 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 22:03:28.118790 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.117804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.2.165.125:41957)
0928 22:03:28.117848 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.117850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.118189 (+ 339us) server_negotiation.cc:407] Connection header received
0928 22:03:28.118301 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.118302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.118315 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.118345 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.118727 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47174: BlockingRecv error: recv got EOF from 127.0.0.1:47174 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:03:28.137611 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.136569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.2.165.125:41957)
0928 22:03:28.136618 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.136620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.136949 (+ 329us) server_negotiation.cc:407] Connection header received
0928 22:03:28.137062 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.137064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.137076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.137107 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.137549 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47188: BlockingRecv error: recv got EOF from 127.0.0.1:47188 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 22:03:28.172350 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.171205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.2.165.125:41957)
0928 22:03:28.171255 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.171258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.171584 (+ 326us) server_negotiation.cc:407] Connection header received
0928 22:03:28.171701 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.171702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.171715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.171747 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.172288 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47200: BlockingRecv error: recv got EOF from 127.0.0.1:47200 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:03:28.238662 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.237972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.2.165.125:41957)
0928 22:03:28.238031 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.238033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.238049 (+ 16us) server_negotiation.cc:407] Connection header received
0928 22:03:28.238148 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.238149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.238164 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.238227 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.238573 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47210: BlockingRecv error: recv got EOF from 127.0.0.1:47210 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 22:03:28.369095 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.368013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.2.165.125:41957)
0928 22:03:28.368096 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.368099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.368276 (+ 177us) server_negotiation.cc:407] Connection header received
0928 22:03:28.368393 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.368395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.368407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.368508 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.369023 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47224: BlockingRecv error: recv got EOF from 127.0.0.1:47224 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 22:03:28.627627 25076 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:28.626975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.2.165.125:41957)
0928 22:03:28.627045 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:28.627047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:28.627055 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:28.627080 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:28.627081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:28.627090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:28.627161 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:28.627548 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47234: BlockingRecv error: recv got EOF from 127.0.0.1:47234 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:03:28.979274 25179 catalog_manager.cc:1259] Loaded cluster ID: 65d454a58b08454dbdb60b0ac89a1743
I20260928 22:03:28.979362 25179 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: loading cluster ID for follower catalog manager: success
I20260928 22:03:28.980432 25179 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: acquiring CA information for follower catalog manager: success
I20260928 22:03:28.980953 25179 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5b9a38511fca42b1a3fd64044a3d0246: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:03:29.025394 25208 catalog_manager.cc:1259] Loaded cluster ID: 65d454a58b08454dbdb60b0ac89a1743
I20260928 22:03:29.025486 25208 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: loading cluster ID for follower catalog manager: success
I20260928 22:03:29.026304 25208 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: acquiring CA information for follower catalog manager: success
I20260928 22:03:29.026692 25208 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c1f73ce1749429f90fc7fbb5b614c9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:03:29.143497 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:29.140968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.2.165.125:41957)
0928 22:03:29.141084 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:29.141086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:29.141103 (+ 17us) server_negotiation.cc:407] Connection header received
0928 22:03:29.141133 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:29.141134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:29.141183 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:29.141280 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:29.141451 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.141953 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.142751 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.142971 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.143031 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:29.143188 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:29.143302 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:29.143361 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:29.143404 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 22:03:29.143423 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260928 22:03:29.148430 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:29.145695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.2.165.124:39355)
0928 22:03:29.145823 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:29.145826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:29.145909 (+ 83us) server_negotiation.cc:407] Connection header received
0928 22:03:29.146013 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:29.146014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:29.146070 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:29.146156 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:29.146267 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.146804 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.147570 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.147793 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.147878 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:29.148070 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:29.148190 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:29.148276 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:29.148301 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 22:03:29.148323 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260928 22:03:29.153246 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:29.150454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.2.165.124:39355)
0928 22:03:29.150500 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:29.150502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:29.150695 (+ 193us) server_negotiation.cc:407] Connection header received
0928 22:03:29.150773 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:29.150774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:29.150784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:29.150829 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:29.151040 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.151755 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.152532 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.152836 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.152899 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:29.153009 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:29.153041 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:29.153098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:29.153131 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 22:03:29.153156 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:03:29.153779 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:29.150594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.2.165.125:41957)
0928 22:03:29.150644 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:29.150645 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:29.150747 (+ 102us) server_negotiation.cc:407] Connection header received
0928 22:03:29.150874 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:29.150875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:29.150889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:29.150919 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:29.151143 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.151640 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.152809 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.153137 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.153200 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:29.153462 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:29.153500 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:29.153552 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:29.153660 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 22:03:29.153688 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:03:29.154287 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:29.150512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.2.165.126:40175)
0928 22:03:29.150617 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:29.150619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:29.150629 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:03:29.150676 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:29.150676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:29.150736 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:29.150829 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:29.150959 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.151746 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.153244 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:29.153544 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:29.153614 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:29.153843 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:29.153983 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:29.154042 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:29.154187 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 22:03:29.154213 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
I20260928 22:03:29.156327 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 24929
I20260928 22:03:29.162305 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 25004
I20260928 22:03:29.167802 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 25079
2026-09-28T22:03:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2881 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:03:29 dist-test-slave-3t36 krb5kdc[25295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:03:29 dist-test-slave-3t36 krb5kdc[25295](info): set up 2 sockets
Sep 28 22:03:29 dist-test-slave-3t36 krb5kdc[25295](info): commencing operation
krb5kdc: starting...
W20260928 22:03:30.225574 2709 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/test-user.keytab.
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.165.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
Entry for principal kudu/127.2.165.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab.
I20260928 22:03:30.358163 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45153
--webserver_interface=127.2.165.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45153,127.2.165.125:37835,127.2.165.124:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:03:30.445597 25308 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:03:30.445822 25308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:30.445848 25308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:30.445870 25308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:30.447000 25308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:30.447036 25308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:30.447050 25308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:30.447062 25308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:30.447074 25308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:30.448532 25308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45153,127.2.165.125:37835,127.2.165.124:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.126.keytab
--principal=kudu/127.2.165.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.126:45153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.165.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:30.448824 25308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:30.449033 25308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:30.454159 25316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:30.454142 25314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:30.454208 25313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:30.454386 25308 server_base.cc:1034] running on GCE node
I20260928 22:03:30.454629 25308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:30.454934 25308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:30.456072 25308 hybrid_clock.cc:648] HybridClock initialized: now 1790633010456048 us; error 40 us; skew 500 ppm
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:30.465716 25308 init.cc:377] Logged in from keytab as kudu/127.2.165.126@KRBTEST.COM (short username kudu)
I20260928 22:03:30.466354 25308 webserver.cc:460] Webserver started at http://127.2.165.126:37787/ using document root <none> and password file <none>
I20260928 22:03:30.466604 25308 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:30.466652 25308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:30.466806 25308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:30.467973 25308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data/instance:
uuid: "e7969b32332f4032bddede945d244144"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "7ed1c0254b7ee71bcc601a5deb524033"
server_key_iv: "f401f2e80f819a968211fbb78e85ffd9"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.468354 25308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/wal/instance:
uuid: "e7969b32332f4032bddede945d244144"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "7ed1c0254b7ee71bcc601a5deb524033"
server_key_iv: "f401f2e80f819a968211fbb78e85ffd9"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.469856 25308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:30.470698 25323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:30.471047 25308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:30.471166 25308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/wal
uuid: "e7969b32332f4032bddede945d244144"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "7ed1c0254b7ee71bcc601a5deb524033"
server_key_iv: "f401f2e80f819a968211fbb78e85ffd9"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.471249 25308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:30.489848 25308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:30.492089 25308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:30.492261 25308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:30.496503 25308 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.126:45153
I20260928 22:03:30.496541 25375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.126:45153 every 8 connection(s)
I20260928 22:03:30.496961 25308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/data/info.pb
I20260928 22:03:30.497984 25376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:30.499624 25376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:30.504081 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 25308
I20260928 22:03:30.504182 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-0/wal/instance
I20260928 22:03:30.504377 2709 external_mini_cluster.cc:1428] Setting key 54fbea0f6154cd31e64a3077c1786a19
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:30.510630 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.500467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.2.165.126:45153)
0928 22:03:30.500586 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.500595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.500723 (+ 128us) server_negotiation.cc:407] Connection header received
0928 22:03:30.501389 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.501391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.501416 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.501498 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.502053 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.502960 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.503872 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.504375 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.504486 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.506694 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.506700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.506700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.506707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.508479 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.509044 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.509048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.509048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.509138 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.509464 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.509465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.509466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.509706 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.509846 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.510157 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.510336 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
I20260928 22:03:30.510634 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.500310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45153 (local address 127.0.0.1:49116)
0928 22:03:30.500528 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.500579 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.501225 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.501521 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.501524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.501808 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.501970 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.501974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.503057 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.503058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.503785 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.503787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.504504 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.504506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.504569 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.505153 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.505162 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.506614 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.508575 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.508584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.508587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.508932 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.509218 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.509218 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.509219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.509345 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.509841 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.509847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.509930 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.510032 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.510237 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260928 22:03:30.512188 25376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.165.125@KRBTEST.COM; defaulting to no policy
I20260928 22:03:30.512707 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.512535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51474)
0928 22:03:30.512587 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.512639 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:03:30.512851 25324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
I20260928 22:03:30.513207 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.513136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51476)
0928 22:03:30.513160 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.513187 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 22:03:30.513463 25376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:37835: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab.
Entry for principal kudu/127.2.165.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab.
I20260928 22:03:30.533612 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:37835
--webserver_interface=127.2.165.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab
--principal=kudu/127.2.165.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45153,127.2.165.125:37835,127.2.165.124:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:03:30.562791 25376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } attempt: 1
I20260928 22:03:30.563395 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.563252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51482)
0928 22:03:30.563325 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.563363 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 22:03:30.563759 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.563660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51484)
0928 22:03:30.563703 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.563734 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:03:30.563848 25376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:37835: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
W20260928 22:03:30.619279 25383 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:03:30.619557 25383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:30.619588 25383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:30.619632 25383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:30.620920 25383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:30.620998 25383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:30.621021 25383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:30.621042 25383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:30.621070 25383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:30.622961 25383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45153,127.2.165.125:37835,127.2.165.124:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.125.keytab
--principal=kudu/127.2.165.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.125:37835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.165.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:30.623306 25383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:30.623562 25383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:30.628928 25391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:30.628991 25389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:30.628952 25388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:30.629469 25383 server_base.cc:1034] running on GCE node
I20260928 22:03:30.629747 25383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:30.630070 25383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:30.631248 25383 hybrid_clock.cc:648] HybridClock initialized: now 1790633010631237 us; error 51 us; skew 500 ppm
I20260928 22:03:30.631981 25376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } attempt: 2
I20260928 22:03:30.632413 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.632258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51486)
0928 22:03:30.632313 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.632354 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:03:30.632776 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.632702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51500)
0928 22:03:30.632729 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.632754 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 22:03:30.632875 25376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.125:37835: Network error: Client connection negotiation failed: client connection to 127.2.165.125:37835: connect: Connection refused (error 111)
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:30.634156 25383 init.cc:377] Logged in from keytab as kudu/127.2.165.125@KRBTEST.COM (short username kudu)
I20260928 22:03:30.634655 25383 webserver.cc:460] Webserver started at http://127.2.165.125:39269/ using document root <none> and password file <none>
I20260928 22:03:30.634850 25383 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:30.634889 25383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:30.634986 25383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:30.636031 25383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data/instance:
uuid: "b4f59d4a86bc47aabedfd54f46eba191"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "1c8c228a263bae7d2a99c0261d4739e5"
server_key_iv: "5a5597a629fdc43fe2aed976d61462f8"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.636346 25383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/wal/instance:
uuid: "b4f59d4a86bc47aabedfd54f46eba191"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "1c8c228a263bae7d2a99c0261d4739e5"
server_key_iv: "5a5597a629fdc43fe2aed976d61462f8"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.637835 25383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:30.638628 25398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:30.638859 25383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:03:30.638957 25383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/wal
uuid: "b4f59d4a86bc47aabedfd54f46eba191"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "1c8c228a263bae7d2a99c0261d4739e5"
server_key_iv: "5a5597a629fdc43fe2aed976d61462f8"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.639070 25383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:30.647869 25383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:30.650440 25383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:30.650609 25383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:30.655426 25383 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.125:37835
I20260928 22:03:30.655473 25450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.125:37835 every 8 connection(s)
I20260928 22:03:30.655853 25383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/data/info.pb
I20260928 22:03:30.656976 25451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:30.658689 25451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:30.659395 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 25383
I20260928 22:03:30.659511 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-1/wal/instance
I20260928 22:03:30.659740 2709 external_mini_cluster.cc:1428] Setting key 36a608a00c11845700b3ea0c376d13cf
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.165.124@KRBTEST.COM; defaulting to no policy
I20260928 22:03:30.670645 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.659637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.2.165.126:45153)
0928 22:03:30.659690 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.659692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.659979 (+ 287us) server_negotiation.cc:407] Connection header received
0928 22:03:30.660667 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.660668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.660715 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.660787 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.661832 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.662596 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.664256 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.664472 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.664537 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.667286 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.667288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.667288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.667297 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.668917 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.669327 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.669328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.669329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.669412 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.669706 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.669707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.669707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.669834 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.669910 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.670445 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.670571 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:03:30.670691 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.659594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45153 (local address 127.0.0.1:49124)
0928 22:03:30.659795 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.659846 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.660543 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.660946 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.660951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.661417 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.661645 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.661650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.662704 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.662706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.664145 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.664149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.664582 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.664583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.664807 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.665410 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.665431 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.667176 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.669006 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.669017 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.669020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.669250 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.669489 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.669490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.669491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.669613 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.669927 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.669934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.670249 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.670365 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.670514 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.165.124@KRBTEST.COM" created.
I20260928 22:03:30.671384 25451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:03:30.679436 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.671685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51506)
0928 22:03:30.671741 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.671754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.671804 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.672052 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.672053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.672199 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.672284 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.672286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.673285 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.673286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.673788 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.673790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.674363 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.674364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.674429 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.674938 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.674942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.676268 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.678212 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.678213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.678213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.678383 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.678642 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.678643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.678643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.678733 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.679031 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.679032 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.679070 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.679135 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.679247 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:03:30.679454 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.671732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.2.165.125:37835)
0928 22:03:30.671902 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.671907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.671933 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:03:30.671957 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.671958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.671970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.672048 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.672358 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.673169 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.673882 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.674303 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.674369 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.676377 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.676388 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.676389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.676401 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.678100 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.678477 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.678480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.678481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.678536 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.678796 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.678797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.678797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.678961 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.679056 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.679163 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.679298 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 22:03:30.680290 25451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:30.680721 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.680577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56134)
0928 22:03:30.680611 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.680658 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 22:03:30.680867 25399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
I20260928 22:03:30.681334 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.681254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56136)
0928 22:03:30.681286 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.681313 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 22:03:30.681474 25451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:37111: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.165.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab.
Entry for principal kudu/127.2.165.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab.
I20260928 22:03:30.689085 2709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjhUuNU/build/release/bin/kudu
/tmp/dist-test-taskjhUuNU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:37111
--webserver_interface=127.2.165.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab
--principal=kudu/127.2.165.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.165.84:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.165.126:45153,127.2.165.125:37835,127.2.165.124:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:03:30.724042 25376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } attempt: 3
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:03:30.730684 25451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } attempt: 1
I20260928 22:03:30.731101 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.730961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56138)
0928 22:03:30.731027 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.731065 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:03:30.731554 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.731448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56140)
0928 22:03:30.731496 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.731525 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:03:30.731815 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.724532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.2.165.125:37835)
0928 22:03:30.724601 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.724603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.724614 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:03:30.724698 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.724699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.724717 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.724779 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.725114 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.725622 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.726408 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.726632 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.726710 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.728931 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.728932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.728933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.728945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.730388 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.730912 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.730913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.730914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.730977 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.731298 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.731299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.731300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.731445 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.731524 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.731686 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.731766 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:03:30.731861 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.724459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51512)
0928 22:03:30.724530 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.724546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.724617 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.724788 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.724789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.724958 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.725025 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.725027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.725718 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.725719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.726313 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.726316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.726707 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.726708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.726794 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.727322 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.727327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.728820 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.730537 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.730538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.730539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.730747 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.731074 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.731075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.731075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.731152 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.731530 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.731531 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.731589 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.731658 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.731767 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 22:03:30.731994 25451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:37111: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
I20260928 22:03:30.732292 25376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:30.732628 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.732521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56152)
0928 22:03:30.732568 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.732603 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 22:03:30.733022 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.732931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56156)
0928 22:03:30.732973 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.732998 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:03:30.733117 25376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:37111: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
I20260928 22:03:30.764284 25376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } attempt: 1
I20260928 22:03:30.764829 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.764715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56170)
0928 22:03:30.764761 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.764798 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 22:03:30.765223 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.765085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56176)
0928 22:03:30.765149 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.765193 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 22:03:30.765380 25376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:37111: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
W20260928 22:03:30.774494 25458 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:03:30.774755 25458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:03:30.774860 25458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:03:30.774897 25458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:03:30.776098 25458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:03:30.776144 25458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:03:30.776158 25458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:03:30.776171 25458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:03:30.776183 25458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:03:30.777815 25458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.165.84:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.165.126:45153,127.2.165.125:37835,127.2.165.124:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/krb5kdc/kudu_127.2.165.124.keytab
--principal=kudu/127.2.165.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.165.124:37111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.165.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 22:03:30.778079 25458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:03:30.778321 25458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:03:30.783205 25463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:30.783232 25464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:03:30.783349 25466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:03:30.783923 25458 server_base.cc:1034] running on GCE node
I20260928 22:03:30.784168 25458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:03:30.784479 25458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:03:30.785674 25458 hybrid_clock.cc:648] HybridClock initialized: now 1790633010785654 us; error 31 us; skew 500 ppm
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:03:30.788537 25458 init.cc:377] Logged in from keytab as kudu/127.2.165.124@KRBTEST.COM (short username kudu)
I20260928 22:03:30.789114 25458 webserver.cc:460] Webserver started at http://127.2.165.124:45591/ using document root <none> and password file <none>
I20260928 22:03:30.789373 25458 fs_manager.cc:359] Metadata directory not provided
I20260928 22:03:30.789453 25458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:03:30.789600 25458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:03:30.790699 25458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data/instance:
uuid: "7b9e25fc25bd487c9677a0ac752b442b"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "db81cb88bb52d7e6f140f4317ebc6e0e"
server_key_iv: "2e4ff120879173646b55f71cf6dffca0"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.791076 25458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/wal/instance:
uuid: "7b9e25fc25bd487c9677a0ac752b442b"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "db81cb88bb52d7e6f140f4317ebc6e0e"
server_key_iv: "2e4ff120879173646b55f71cf6dffca0"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.792505 25458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:03:30.793432 25473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:03:30.793609 25458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:03:30.793717 25458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data,/tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/wal
uuid: "7b9e25fc25bd487c9677a0ac752b442b"
format_stamp: "Formatted at 2026-09-28 22:03:30 on dist-test-slave-3t36"
server_key: "db81cb88bb52d7e6f140f4317ebc6e0e"
server_key_iv: "2e4ff120879173646b55f71cf6dffca0"
server_key_version: "encryptionkey@0"
I20260928 22:03:30.793803 25458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:03:30.800144 25451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } attempt: 2
I20260928 22:03:30.800599 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.800467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56186)
0928 22:03:30.800530 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.800569 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:03:30.801014 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.800924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56194)
0928 22:03:30.800968 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.800992 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:03:30.801100 25451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.165.124:37111: Network error: Client connection negotiation failed: client connection to 127.2.165.124:37111: connect: Connection refused (error 111)
I20260928 22:03:30.803645 25458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:03:30.805646 25458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:03:30.805815 25458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:03:30.809926 25458 rpc_server.cc:307] RPC server started. Bound to: 127.2.165.124:37111
I20260928 22:03:30.809978 25525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.165.124:37111 every 8 connection(s)
I20260928 22:03:30.810369 25458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/data/info.pb
I20260928 22:03:30.811468 25526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:03:30.813131 25526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:30.814357 2709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjhUuNU/build/release/bin/kudu as pid 25458
I20260928 22:03:30.814471 2709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjhUuNU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632730261496-2709-0/minicluster-data/master-2/wal/instance
I20260928 22:03:30.814697 2709 external_mini_cluster.cc:1428] Setting key f1abe1a29178fdccdb6ade1b54964424
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:03:30.824182 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.813896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.2.165.126:45153)
0928 22:03:30.813969 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.813972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.814254 (+ 282us) server_negotiation.cc:407] Connection header received
0928 22:03:30.814653 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.814654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.814668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.814729 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.815696 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.816447 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.817739 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.818006 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.818094 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.820709 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.820710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.820711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.820718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.822355 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.822902 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.822904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.822904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.822975 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.823319 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.823320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.823320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.823439 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.823489 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.823951 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.824068 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:03:30.824275 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.813791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.126:45153 (local address 127.0.0.1:49128)
0928 22:03:30.813962 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.814005 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.814510 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.814897 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.814903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.815309 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.815605 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.815610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.816523 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.816524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.817646 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.817648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.818132 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.818134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.818252 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.818825 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.818834 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.820589 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.822442 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.822452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.822454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.822797 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.823061 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.823062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.823063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.823231 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.823512 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.823518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.823689 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.823885 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.824061 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260928 22:03:30.824929 25526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:30.825459 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.815110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.2.165.124:37111)
0928 22:03:30.815277 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.815282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.815333 (+ 51us) server_negotiation.cc:407] Connection header received
0928 22:03:30.815407 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.815409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.815428 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.815515 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.816224 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.817459 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.818611 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.818947 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.819031 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.821860 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.821869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.821870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.821877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.823945 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.824380 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.824383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.824384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.824448 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.824696 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.824697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.824697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.824938 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.825081 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.825216 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.825334 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.126@KRBTEST.COM
I20260928 22:03:30.832432 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.825220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.125:37835 (local address 127.0.0.1:51520)
0928 22:03:30.825297 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.825308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.825345 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.825729 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.825730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.825841 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.825928 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.825931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.826679 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.826681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.827178 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.827179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.827581 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.827582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.827655 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.828136 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.828141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.829451 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.831112 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.831113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.831114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.831289 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.831515 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.831515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.831516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.831780 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.832085 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.832086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.832136 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.832220 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.832325 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 22:03:30.832444 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.825283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.2.165.125:37835)
0928 22:03:30.825331 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.825333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.825341 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:30.825482 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.825483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.825492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.825560 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.826032 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.826591 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.827267 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.827498 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.827580 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.829547 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.829548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.829549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.829557 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.831028 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.831382 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.831384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.831384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.831444 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.831880 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.831881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.831882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.832015 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.832094 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.832285 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.832362 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:03:30.832857 25526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:03:30.834095 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.826271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.2.165.126:45153)
0928 22:03:30.826326 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.826328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.826335 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:03:30.826355 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.826355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.826364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.826409 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.826887 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.827349 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.828136 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.828473 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.828559 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.831158 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.831160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.831161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.831169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.832740 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.833097 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.833099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.833099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.833239 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.833522 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.833523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.833523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.833625 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.833827 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.833919 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.833996 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.124@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.165.125@KRBTEST.COM
I20260928 22:03:30.839661 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.833125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56200)
0928 22:03:30.833173 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.833183 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.833220 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.833348 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.833349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.833503 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.833545 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.833547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.834190 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.834191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.834863 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.834865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.835238 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.835239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.835293 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.835691 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.835696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.836847 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.838445 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.838445 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.838446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.838646 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.838913 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.838914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.838915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.838977 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.839314 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.839315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.839362 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.839446 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.839543 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:03:30.839787 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.833176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.2.165.124:37111)
0928 22:03:30.833212 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.833215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.833223 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:30.833282 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.833282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.833293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.833345 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.833600 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.834110 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.834949 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.835159 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.835240 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.836925 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.836927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.836927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.836934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.838346 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.838730 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.838731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.838731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.838837 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.839067 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.839068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.839091 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.839234 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.839300 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.839548 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.839643 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:03:30.840548 25376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } attempt: 2
I20260928 22:03:30.841744 25526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: Bootstrap starting.
I20260928 22:03:30.842813 25526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:30.843119 25526 log.cc:824] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:30.843302 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.834814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.2.165.125:37835)
0928 22:03:30.834866 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.834868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.834887 (+ 19us) server_negotiation.cc:407] Connection header received
0928 22:03:30.834929 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.834930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.834939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.834986 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.835463 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.835912 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.836617 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.836925 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.836961 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.839554 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.839556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.839556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.839564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.841706 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.842218 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.842219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.842220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.842299 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.842550 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.842551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.842552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.842677 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.842975 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.843019 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.843108 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:03:30.844092 25526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: No bootstrap required, opened a new log
I20260928 22:03:30.845249 2709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.126@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:03:30.846678 25526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.846868 25526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9e25fc25bd487c9677a0ac752b442b, State: Initialized, Role: FOLLOWER
I20260928 22:03:30.846985 25526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.846239 25350 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e7969b32332f4032bddede945d244144"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49134
I20260928 22:03:30.847306 25532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.847364 25526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:30.847472 25532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:30.849105 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.840855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56212)
0928 22:03:30.840911 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.840923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.840964 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.841216 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.841217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.841385 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.841430 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.841431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.842444 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.842445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.843261 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.843264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.844013 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.844014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.844082 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.844499 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.844505 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.845985 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.847853 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.847854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.847855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.848094 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.848306 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.848306 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.848307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.848367 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.848781 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.848782 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.848838 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.848912 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.849014 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:03:30.849105 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.840901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.2.165.124:37111)
0928 22:03:30.841076 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.841078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.841086 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:30.841110 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.841111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.841119 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.841194 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.841509 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.842373 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.843593 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.843888 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.843969 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.846058 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.846059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.846060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.846067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.847755 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.848175 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.848176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.848177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.848235 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.848442 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.848444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.848444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.848660 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.848784 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.848947 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.849017 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260928 22:03:30.849889 25543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:30.850003 25543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:30.850838 25376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144: Bootstrap starting.
I20260928 22:03:30.851560 25376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:30.851873 25376 log.cc:824] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:30.853024 25376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144: No bootstrap required, opened a new log
I20260928 22:03:30.849967 25458 master_runner.cc:186] Error getting master registration for 127.2.165.126:45153: OK, instance_id { permanent_uuid: "e7969b32332f4032bddede945d244144" instance_seqno: 1790633010495244 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:03:30.853401 25458 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:03:30.854391 25376 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.854539 25376 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7969b32332f4032bddede945d244144, State: Initialized, Role: FOLLOWER
I20260928 22:03:30.854629 25376 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.854964 25376 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:30.854929 25546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.854990 25350 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:03:30.855069 25350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:30.855888 25350 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.856149 25350 leader_election.cc:288] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [CANDIDATE]: Term 1 election: Requested vote from peers b4f59d4a86bc47aabedfd54f46eba191 (127.2.165.125:37835), 7b9e25fc25bd487c9677a0ac752b442b (127.2.165.124:37111)
I20260928 22:03:30.856602 25500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7969b32332f4032bddede945d244144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b9e25fc25bd487c9677a0ac752b442b"
I20260928 22:03:30.856757 25500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:30.857419 25546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:30.857836 25500 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7969b32332f4032bddede945d244144 in term 1.
I20260928 22:03:30.858134 25324 leader_election.cc:302] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b9e25fc25bd487c9677a0ac752b442b, e7969b32332f4032bddede945d244144; no voters:
I20260928 22:03:30.858409 25546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:03:30.858687 25546 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [term 1 LEADER]: Becoming Leader. State: Replica: e7969b32332f4032bddede945d244144, State: Running, Role: LEADER
I20260928 22:03:30.858867 25546 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.859194 25308 master_runner.cc:186] Error getting master registration for 127.2.165.125:37835: OK, instance_id { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" instance_seqno: 1790633010654047 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:03:30.862484 25308 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:03:30.862703 25564 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7969b32332f4032bddede945d244144. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.862826 25564 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:30.863121 25567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:03:30.864017 25567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:03:30.869601 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.858156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.2.165.126:45153)
0928 22:03:30.858416 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.858418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.860163 (+ 1745us) server_negotiation.cc:407] Connection header received
0928 22:03:30.860346 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.860347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.860360 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.860440 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.861408 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.862161 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.863334 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.863673 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.863727 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.865547 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.865548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.865549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.865557 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.867311 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.867771 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.867772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.867773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.867850 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.868066 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.868068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.868068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.868203 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.868296 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.869394 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.869485 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260928 22:03:30.872421 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.857850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.2.165.125:37835)
0928 22:03:30.857901 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.857903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.863095 (+ 5192us) server_negotiation.cc:407] Connection header received
0928 22:03:30.863204 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.863205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.863217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.863252 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.864093 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.864906 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.866099 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.866490 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.866544 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.868146 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.868148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.868148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.868157 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.870577 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.871133 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.871134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.871135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.871218 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.871437 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.871454 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.871455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.871905 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.872017 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.872224 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.872321 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:03:30.876959 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.858349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.2.165.124:37111)
0928 22:03:30.858410 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.858412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.864825 (+ 6413us) server_negotiation.cc:407] Connection header received
0928 22:03:30.864934 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.864935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.864946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.864979 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.868118 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.868950 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.870862 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.871213 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.871258 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.873392 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.873394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.873394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.873403 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.875454 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.875923 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.875925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.875925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.875994 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.876224 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.876225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.876225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.876389 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.876470 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.876758 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.876849 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:03:30.882053 25500 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [term 1 FOLLOWER]: Refusing update from remote peer e7969b32332f4032bddede945d244144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:30.882447 25546 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:30.884291 25532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7969b32332f4032bddede945d244144. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.884407 25532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:30.886139 25564 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.886245 25564 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: This master's current role is: LEADER
I20260928 22:03:30.886333 25532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.886422 25532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:30.886535 25569 mvcc.cc:205] Tried to move back new op lower bound from 7334432812568899584 to 7334432812479381504. Current Snapshot: MvccSnapshot[applied={T|T < 7334432812568899584}]
I20260928 22:03:30.886624 25570 mvcc.cc:205] Tried to move back new op lower bound from 7334432812568899584 to 7334432812479381504. Current Snapshot: MvccSnapshot[applied={T|T < 7334432812568899584}]
I20260928 22:03:30.887404 25567 catalog_manager.cc:1347] Generated new cluster ID: e5b7c79d6d784809a0cab23f548ba237
I20260928 22:03:30.887462 25567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:03:30.892167 25451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } attempt: 3
Sep 28 22:03:30 dist-test-slave-3t36 krb5kdc[25295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790633010, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.165.125@KRBTEST.COM for kudu/127.2.165.124@KRBTEST.COM
I20260928 22:03:30.901392 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.892458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.165.124:37111 (local address 127.0.0.1:56222)
0928 22:03:30.892577 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 22:03:30.892589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:03:30.892656 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:03:30.892966 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:03:30.892968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:03:30.893160 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:03:30.893217 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.893219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.894133 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.894134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.894989 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:03:30.894991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.895525 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.895526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:03:30.895590 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.896551 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:03:30.896556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:03:30.898284 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:03:30.900221 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.900222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.900223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.900451 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.900659 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:03:30.900660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:03:30.900660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:03:30.900747 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:03:30.901045 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:03:30.901046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:03:30.901093 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:03:30.901166 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:03:30.901273 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 22:03:30.901937 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:30.892755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.2.165.124:37111)
0928 22:03:30.892848 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.892850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.892859 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:03:30.892882 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.892883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.892894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.892958 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.893286 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.894054 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.895066 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:30.895386 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:30.895456 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:30.898403 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:03:30.898405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:03:30.898405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:03:30.898414 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:03:30.900120 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.900535 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.900536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.900537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.900593 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:03:30.900813 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:03:30.900814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:03:30.900815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:03:30.900957 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:03:30.901019 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:30.901760 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 22:03:30.901843 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:03:30.904115 25451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: Bootstrap starting.
I20260928 22:03:30.904919 25451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: Neither blocks nor log segments found. Creating new log.
I20260928 22:03:30.905223 25451 log.cc:824] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: Log is configured to *not* fsync() on all Append() calls
I20260928 22:03:30.906527 25567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:03:30.908102 25567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:03:30.909476 25451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: No bootstrap required, opened a new log
I20260928 22:03:30.911618 25451 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.911787 25451 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f59d4a86bc47aabedfd54f46eba191, State: Initialized, Role: FOLLOWER
I20260928 22:03:30.911876 25451 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } }
I20260928 22:03:30.912250 25451 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:03:30.912379 25424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:03:30.913472 25424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [term 1 FOLLOWER]: Refusing update from remote peer e7969b32332f4032bddede945d244144: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:03:30.913774 25425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7969b32332f4032bddede945d244144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4f59d4a86bc47aabedfd54f46eba191"
I20260928 22:03:30.914290 25564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:03:30.914601 25425 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7969b32332f4032bddede945d244144 in term 1.
W20260928 22:03:30.917702 25586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:03:30.917802 25586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:03:30.917924 25567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144: Generated new TSK 0
I20260928 22:03:30.917990 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.918071 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:30.918123 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7969b32332f4032bddede945d244144. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.918174 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:30.918205 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.918253 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:03:30.918506 25567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:03:30.921319 25588 mvcc.cc:205] Tried to move back new op lower bound from 7334432812661182464 to 7334432812479381504. Current Snapshot: MvccSnapshot[applied={T|T < 7334432812568899584}]
I20260928 22:03:30.922286 25546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7969b32332f4032bddede945d244144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7969b32332f4032bddede945d244144" member_type: VOTER last_known_addr { host: "127.2.165.126" port: 45153 } } peers { permanent_uuid: "b4f59d4a86bc47aabedfd54f46eba191" member_type: VOTER last_known_addr { host: "127.2.165.125" port: 37835 } } peers { permanent_uuid: "7b9e25fc25bd487c9677a0ac752b442b" member_type: VOTER last_known_addr { host: "127.2.165.124" port: 37111 } } }
I20260928 22:03:30.922420 25546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7969b32332f4032bddede945d244144 [sys.catalog]: This master's current role is: LEADER
W20260928 22:03:30.952169 25379 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.950476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.2.165.126:45153)
0928 22:03:30.950523 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.950526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.951427 (+ 901us) server_negotiation.cc:407] Connection header received
0928 22:03:30.951537 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.951538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.951550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.951582 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.952085 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49156: BlockingRecv error: recv got EOF from 127.0.0.1:49156 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:03:30.953137 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.950889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.2.165.125:37835)
0928 22:03:30.950929 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.950931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.952571 (+ 1640us) server_negotiation.cc:407] Connection header received
0928 22:03:30.952704 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.952706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.952717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.952748 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.953076 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51540: BlockingRecv error: recv got EOF from 127.0.0.1:51540 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:03:30.954020 25530 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.951174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.2.165.124:37111)
0928 22:03:30.951213 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.951215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.953497 (+ 2282us) server_negotiation.cc:407] Connection header received
0928 22:03:30.953594 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.953595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.953606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.953635 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.953956 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56226: BlockingRecv error: recv got EOF from 127.0.0.1:56226 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:03:30.956739 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.955728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.2.165.125:37835)
0928 22:03:30.955872 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.955874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.956145 (+ 271us) server_negotiation.cc:407] Connection header received
0928 22:03:30.956257 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.956258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.956291 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.956323 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.956662 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51552: BlockingRecv error: recv got EOF from 127.0.0.1:51552 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:03:30.960024 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.959207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.2.165.125:37835)
0928 22:03:30.959245 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.959247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.959464 (+ 217us) server_negotiation.cc:407] Connection header received
0928 22:03:30.959564 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.959565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.959577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.959606 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.959958 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51568: BlockingRecv error: recv got EOF from 127.0.0.1:51568 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 22:03:30.964205 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.963407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.2.165.125:37835)
0928 22:03:30.963446 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.963448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.963653 (+ 205us) server_negotiation.cc:407] Connection header received
0928 22:03:30.963748 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.963749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.963760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.963802 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.964150 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51582: BlockingRecv error: recv got EOF from 127.0.0.1:51582 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:03:30.970671 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.969724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.2.165.125:37835)
0928 22:03:30.969769 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.969771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.970041 (+ 270us) server_negotiation.cc:407] Connection header received
0928 22:03:30.970155 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.970157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.970170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.970203 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.970611 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51594: BlockingRecv error: recv got EOF from 127.0.0.1:51594 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:03:30.981372 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.980337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.2.165.125:37835)
0928 22:03:30.980386 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.980388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.980665 (+ 277us) server_negotiation.cc:407] Connection header received
0928 22:03:30.980832 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.980834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.980848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.980880 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.981307 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51608: BlockingRecv error: recv got EOF from 127.0.0.1:51608 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:03:31.000012 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:30.999014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.2.165.125:37835)
0928 22:03:30.999064 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:30.999066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:30.999365 (+ 299us) server_negotiation.cc:407] Connection header received
0928 22:03:30.999478 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:30.999479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:30.999495 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:30.999528 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:30.999949 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51612: BlockingRecv error: recv got EOF from 127.0.0.1:51612 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 22:03:31.034694 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:31.033670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.2.165.125:37835)
0928 22:03:31.033716 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:31.033719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:31.034042 (+ 323us) server_negotiation.cc:407] Connection header received
0928 22:03:31.034170 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:31.034171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:31.034185 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:31.034217 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:31.034633 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51618: BlockingRecv error: recv got EOF from 127.0.0.1:51618 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:03:31.101212 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:31.100389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.2.165.125:37835)
0928 22:03:31.100444 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:31.100446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:31.100516 (+ 70us) server_negotiation.cc:407] Connection header received
0928 22:03:31.100578 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:31.100579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:31.100589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:31.100655 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:31.101142 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51630: BlockingRecv error: recv got EOF from 127.0.0.1:51630 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 22:03:31.231282 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:31.230456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.2.165.125:37835)
0928 22:03:31.230574 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:31.230576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:31.230590 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:03:31.230672 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:31.230673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:31.230685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:31.230745 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:31.231234 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51634: BlockingRecv error: recv got EOF from 127.0.0.1:51634 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 22:03:31.489254 25455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:03:31.488589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.2.165.125:37835)
0928 22:03:31.488666 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:31.488691 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:31.488703 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:03:31.488782 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:31.488783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:31.488793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:31.488855 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:03:31.489205 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51648: BlockingRecv error: recv got EOF from 127.0.0.1:51648 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:03:31.850522 25543 catalog_manager.cc:1259] Loaded cluster ID: e5b7c79d6d784809a0cab23f548ba237
I20260928 22:03:31.850607 25543 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: loading cluster ID for follower catalog manager: success
I20260928 22:03:31.851478 25543 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: acquiring CA information for follower catalog manager: success
I20260928 22:03:31.851823 25543 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b9e25fc25bd487c9677a0ac752b442b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:03:31.918299 25586 catalog_manager.cc:1259] Loaded cluster ID: e5b7c79d6d784809a0cab23f548ba237
I20260928 22:03:31.918401 25586 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: loading cluster ID for follower catalog manager: success
I20260928 22:03:31.919263 25586 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: acquiring CA information for follower catalog manager: success
I20260928 22:03:31.919611 25586 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4f59d4a86bc47aabedfd54f46eba191: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:03:32.005092 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:32.002496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.2.165.125:37835)
0928 22:03:32.002625 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:32.002627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:32.002637 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:03:32.002698 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:32.002699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:32.002759 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:32.002831 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:32.003025 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.003548 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.004312 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.004509 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.004563 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:32.004763 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:32.004874 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:32.004936 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:32.005008 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 22:03:32.005024 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260928 22:03:32.010020 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:32.007308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.2.165.124:37111)
0928 22:03:32.007423 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:32.007425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:32.007452 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:03:32.007562 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:32.007562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:32.007608 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:32.007697 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:32.007875 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.008386 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.009197 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.009412 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.009487 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:32.009650 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:32.009764 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:32.009844 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:32.009897 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:03:32.009919 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260928 22:03:32.014916 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:32.011917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.2.165.125:37835)
0928 22:03:32.011960 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:32.011962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:32.011971 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:03:32.011999 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:32.012000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:32.012011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:32.012062 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:32.012322 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.013026 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.014110 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.014394 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.014458 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:32.014628 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:32.014665 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:32.014696 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:32.014803 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:03:32.014828 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:03:32.015264 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:32.012121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.2.165.124:37111)
0928 22:03:32.012201 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:32.012203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:32.012212 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:03:32.012316 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:32.012317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:32.012328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:32.012357 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:32.012534 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.013345 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.014278 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.014532 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.014592 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:32.014765 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:32.014808 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:32.014836 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:32.015149 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 22:03:32.015174 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:03:32.015427 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:03:32.011805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.2.165.126:45153)
0928 22:03:32.012135 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 22:03:32.012137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:03:32.012145 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:03:32.012189 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:03:32.012190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:03:32.012243 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:03:32.012320 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:03:32.012483 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.013372 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.014216 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:03:32.014524 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:03:32.014596 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:03:32.014798 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:03:32.014907 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:03:32.014972 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:03:32.015256 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 22:03:32.015294 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":61,"threads_started":1}
I20260928 22:03:32.017318 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 25308
I20260928 22:03:32.022794 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 25383
I20260928 22:03:32.027676 2709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjhUuNU/build/release/bin/kudu with pid 25458
2026-09-28T22:03:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2859 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5741 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281773 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST